README.md
1# Certificate Transparency: Go Code
2
3[](https://travis-ci.org/google/certificate-transparency-go)
4[](https://goreportcard.com/report/github.com/google/certificate-transparency-go)
5[](https://godoc.org/github.com/google/certificate-transparency-go)
6
7This repository holds Go code related to
8[Certificate Transparency](https://www.certificate-transparency.org/) (CT). The
9repository requires Go version 1.9.
10
11 - [Repository Structure](#repository-structure)
12 - [Trillian CT Personality](#trillian-ct-personality)
13 - [Working on the Code](#working-on-the-code)
14 - [Rebuilding Generated Code](#rebuilding-generated-code)
15 - [Updating Vendor Code](#updating-vendor-code)
16 - [Running Codebase Checks](#running-codebase-checks)
17
18## Repository Structure
19
20The main parts of the repository are:
21
22 - Encoding libraries:
23 - `asn1/` and `x509/` are forks of the upstream Go `encoding/asn1` and
24 `crypto/x509` libraries. We maintain separate forks of these packages
25 because CT is intended to act as an observatory of certificates across the
26 ecosystem; as such, we need to be able to process somewhat-malformed
27 certificates that the stricter upstream code would (correctly) reject.
28 Our `x509` fork also includes code for working with the
29 [pre-certificates defined in RFC 6962](https://tools.ietf.org/html/rfc6962#section-3.1).
30 - `tls` holds a library for processing TLS-encoded data as described in
31 [RFC 5246](https://tools.ietf.org/html/rfc5246).
32 - `x509util` provides additional utilities for dealing with
33 `x509.Certificate`s.
34 - CT client libraries:
35 - The top-level `ct` package (in `.`) holds types and utilities for working
36 with CT data structures defined in
37 [RFC 6962](https://tools.ietf.org/html/rfc6962).
38 - `client/` and `jsonclient/` hold libraries that allow access to CT Logs
39 via entrypoints described in
40 [section 4 of RFC 6962](https://tools.ietf.org/html/rfc6962#section-4).
41 - `scanner/` holds a library for scanning the entire contents of an existing
42 CT Log.
43 - Command line tools:
44 - `./client/ctclient` allows interaction with a CT Log
45 - `./scanner/scanlog` allows an existing CT Log to be scanned for certificates
46 of interest; please be polite when running this tool against a Log.
47 - `./x509util/certcheck` allows display and verification of certificates
48 - `./x509util/crlcheck` allows display and verification of certificate
49 revocation lists (CRLs).
50 - CT Personality for [Trillian](https://github.com/google/trillian):
51 - `trillian/` holds code that allows a Certificate Transparency Log to be
52 run using a Trillian Log as its back-end -- see
53 [below](#trillian-ct-personality).
54
55
56## Trillian CT Personality
57
58The `trillian/` subdirectory holds code and scripts for running a CT Log based
59on the [Trillian](https://github.com/google/trillian) general transparency Log.
60
61The main code for the CT personality is held in `trillian/ctfe`; this code
62responds to HTTP requests on the
63[CT API paths](https://tools.ietf.org/html/rfc6962#section-4) and translates
64them to the equivalent gRPC API requests to the Trillian Log.
65
66This obviously relies on the gRPC API definitions at
67`github.com/google/trillian`; the code also uses common libraries from the
68Trillian project for:
69 - exposing monitoring and statistics via an `interface` and corresponding
70 Prometheus implementation (`github.com/google/trillian/monitoring/...`)
71 - dealing with cryptographic keys (`github.com/google/trillian/crypto/...`).
72
73The `trillian/integration/` directory holds scripts and tests for running the whole
74system locally. In particular:
75 - `trillian/integration/ct_integration_test.sh` brings up local processes
76 running a Trillian Log server, signer and a CT personality, and exercises the
77 complete set of RFC 6962 API entrypoints.
78 - `trillian/integration/ct_hammer_test.sh` brings up a complete system and runs
79 a continuous randomized test of the CT entrypoints.
80
81These scripts require a local database instance to be configured as described
82in the [Trillian instructions](https://github.com/google/trillian#mysql-setup).
83
84
85## Working on the Code
86
87Developers who want to make changes to the codebase need some additional
88dependencies and tools, described in the following sections. The
89[Travis configuration](.travis.yml) for the codebase is also useful reference
90for the required tools and scripts, as it may be more up-to-date than this
91document.
92
93### Rebuilding Generated Code
94
95Some of the CT Go code is autogenerated from other files:
96
97 - [Protocol buffer](https://developers.google.com/protocol-buffers/) message
98 definitions are converted to `.pb.go` implementations.
99 - A mock implementation of the Trillian gRPC API (in `trillian/mockclient`) is
100 created with [GoMock](https://github.com/golang/mock).
101
102Re-generating mock or protobuffer files is only needed if you're changing
103the original files; if you do, you'll need to install the prerequisites:
104
105 - `mockgen` tool from https://github.com/golang/mock
106 - `protoc`, [Go support for protoc](https://github.com/golang/protobuf) (see
107 documentation linked from the
108 [protobuf site](https://github.com/google/protobuf))
109
110and run the following:
111
112```bash
113go generate -x ./... # hunts for //go:generate comments and runs them
114```
115
116### Updating Vendor Code
117
118The codebase includes a couple of external projects under the `vendor/`
119subdirectory, to ensure that builds use a fixed version (typically because the
120upstream repository does not guarantee back-compatibility between the tip
121`master` branch and the current stable release). See
122[instructions in the Trillian repo](https://github.com/google/trillian#updating-vendor-code)
123for how to update vendored subtrees.
124
125
126### Running Codebase Checks
127
128The [`scripts/presubmit.sh`](scripts/presubmit.sh) script runs various tools
129and tests over the codebase.
130
131```bash
132# Install gometalinter and all linters
133go get -u github.com/alecthomas/gometalinter
134gometalinter --install
135
136# Run code generation, build, test and linters
137./scripts/presubmit.sh
138
139# Run build, test and linters but skip code generation
140./scripts/presubmit.sh --no-generate
141
142# Or just run the linters alone:
143gometalinter --config=gometalinter.json ./...
144```
145
){kind=link}
){kind=link}