Jake Donham > Technical Difficulties > Read the Code: GitHub Flat Viewer

Read the Code: GitHub Flat Viewer

2022-01-26

One way I like to learn new a computer thing—a new library, language, tool, or technique—is to read the code of projects that use it. I don't do this enough! I usually start by following a "getting started" tutorial, skimming the documentation, and looking at example code. But often the distance between getting started with a computer thing and using it effectively is very large. So reading the code of a substantial project can be really helpful: it's working code with a purpose; you can see how the new thing works and how it fits together with other things. I also like reading project code because I usually learn a lot of things beyond what I set out to learn, like tools and libraries I hadn't heard of, or programming styles and idioms.

I'm starting a new job soon at GitHub Next, so I thought I would read the code of some existing GitHub Next projects to get a sense of how the team works. I started with Flat Viewer, a browser app that reads flat data files from GitHub via the REST API and displays them in a table interface with sorting and filtering. I think it's a pretty straightforward browser app, but I am not a very experienced browser app programmer, so I learned a lot. In what follows I'm going to focus on the parts that are new to me, so this will be a bit of a hodgepodge, but I hope there's something in it that's new and/or useful for you.

The world of libraries, frameworks, and tools for building browser apps is immense and ever-changing—"best practice" is a destination forever receding into the distance as you approach—so please read this as a description of one way of doing things, that seems to work for a particular purpose, as one person understands it, rather than anything like an authoritative explanation.

Flat Viewer is a React app written in TypeScript. Good, I feel pretty comfortable with this! Let's walk through the components:

App

The main component is App, and the main thing it does is route to different views within the app based on the URL path, using React Router. In a single-page React app, navigating to a view doesn't interact with the browser's URL bar: history is not recorded, and you can't get back to a view by copying and pasting its URL. React Router parses the URL and routes to a matching view; navigating to a view updates the URL (and records it in history). This works using a pattern-matching syntax inside of React JSX tags, like so:

<Router>
  <Switch>
    <Route exact path="/" component={Home} />
    <Route exact path="/:org/" component={OrgListing} />
    <Route path="/:owner/:name" component={RepoDetail} />
  </Switch>
</Router>

An empty path / routes to the Home component; a one-segment path like /githubocto/ routes to the OrgListing component; a two-segment path like /githubocto/flat-demo-bitcoin-price routes to the RepoDetail component.

There are a few other things going on in App:

show a progress bar while data is being fetched using NProgress
set up a React context for react-query for reading data from the GitHub API (this is actually one level outside App in main.tsx, so the progress bar in App can use it)
set up a context for react-head, so view components can set the page title with a local <Title> tag (instead of twiddling document.title)
set up a context for use-query-params, which is used to store component state in query params (so browser history and copying/pasting the URL works for that state as well as view navigation; more on this below)

Home and RepoForm

The Home component just adds some styling around RepoForm, which shows a form for filling in a GitHub account and repo. There are several interesting things going on in RepoForm:

the form is rendered using Formik, which keeps track of the state of field values, validates the values, and prevents submitting the form if any values are invalid.
field values are validated using yup, which takes a schema describing expected values, like
```
const validationSchema = object().shape({
  owner: string().required("Please enter a repository owner"),
  name: string().optional(),
});
```
It's integrated with Formik so you can pass the schema directly into the <Formik> tag.
submitting the form adds a new path to the browser history using React Router's useHistory API; this causes the <Router> component in Home to re-render, changing the view. (RepoForm also uses Link to link to an example repo, which does the same thing.)
fields are styled differently when they are invalid; this is handled using classcat, which builds a string of CSS class names from an object mapping names to flags. (More below on styling components.)

There's always a tradeoff between calling out to a dependency and writing the code by hand—I would probably write something simple like this by hand (keep track of field values with React.useState and validate them inline). But I can see that for a big form with intricate validation, using Formik and yup would be much easier; and if they are in your toolbox already it's easier to use them even for simple things.

OrgListing

The OrgListing component displays a list of repos from a GitHub account. The list of repos is queried from the GitHub REST API using wretch and React Query.

Wretch is a wrapper around the browser built-in fetch function (which makes an HTTP request and returns a Promise of the result). It replaces fetch's "big bag of options"-style interface with a "builder"-style interface, and provides some prebuilt handlers for HTTP-level error responses and for decoding response bodies. Here's the wretch call to get a list of repos (see fetchOrgRepos):

githubWretch
  .url(`/search/repositories`)
  .query({ q: `topic:flat-data org:${orgName}`, per_page: 100 })
  .get()
  .json();

It calls the API (at https://api.github.com; this is pre-configured in githubWretch) with the given path and query params (see the API docs), then decodes a successful response as JSON. (Note that the search matches only repos tagged flat-data.)

React rendering is basically synchronous, but wretch returns an asynchronous Promise. This is where React Query comes in: it keeps track of the state of a query—succeeded (with some value), failed (with some error), or waiting—and rerenders components that depend on the query when the state changes (e.g. the Promise resolves). To query the list of repos, the wretch query above is wrapped in a useQuery hook (in useOrgFlatRepos):

useQuery(
  ["org", orgName],
  () => fetchOrgRepos(orgName),
  { retry: false, refetchOnWindowFocus: false }
)

The return value of useQuery includes a status field of type "success" | "error" | "loading"; depending on status it has additional fields describing the returned value or error. OrgListing renders an appropriate component (RepoListing, ErrorState, or Spinner) depending on status. RepoListing renders a list of repos, each with a React Router Link to navigate to the RepoDetail view.

The arguments to useQuery are a label (used for caching and deduplicating queries), a Promise-returning function that makes the underlying query (you can use any library you like), and some options controlling when the query should be retried or refetched (by default, focusing away then returning to the browser refetches the query).

RepoDetail

The RepoDetail component displays controls to select a data file and version from a repo, and a table view of the selected data. By default the first file (having an appropriate format) in the repo and latest version are shown.

The state representing the selected data file and version is tracked with use-query-params. This provides a useQueryParam hook that works like React.useState:

const [filename, setFilename] = useQueryParam("filename", StringParam);
const [selectedSha, setSelectedSha] = useQueryParam("sha", StringParam);

but serializes the state value (StringParam specifies how to serialize it) and stores it in URL query params. It integrates with React Router; updating a state variable (e.g. calling setFilename) causes a navigation, so the state is restored if you copy/paste the URL or press the back button. Nice! There is a lot I don't understand about how React Router and use-query-params work under the hood—it would be worth reading this code, but not today.

When RepoDetail renders, it first queries a list of files in the repo (with useGetFiles and fetchFilesFromRepo; see the API docs) and sets filename to the first valid filename. Next it queries the list of commits for the file (with useCommits and fetchCommits; see the API docs) and sets selectedSha to the hash of the latest commit. React Query has a nice way to chain asynchronous queries: useQuery supports an enabled flag, which the useCommits query sets only when the useGetFiles query has completed successfully.

UseCommits / fetchCommits refer to the type of the API response using Endpoints from @octokit/types:

type listCommitsResponse =
  Endpoints["GET /repos/{owner}/{repo}/commits"]["response"];

This is very cool! Apparently the types are scraped from the API docs. It looks like Octokit doesn't have full types for the search API (used in OrgListing), so those types are written out explicitly. (I wonder why Flat Viewer doesn't use the Octokit client.)

Commits of data files created by Flat Action contain the URL of the data source (see example); RepoDetail parses it out (with parseFlatCommitMessage) in order to display a link.

Finally, RepoDetail renders its controls (with components to show file and version pickers) and the data file itself (using JSONDetail). There's also a "toast" component for displaying error messages, and a "disclosure" component that makes the controls hideable on smaller displays; I didn't dig into the details.

JSONDetail

The JSONDetail component queries the contents of a data file (with useDataFile and fetchDataFile) and displays the contents using flat-ui. It also displays a picker for keys within the file—I guess it handles JSON files with a top-level object containing multiple arrays? FetchDataFile parses the file as CSV, TSV, YAML, or JSON based on the file extension (using d3-dsv and yaml), and has some special handling for GeoJSON / TopoJSON files.

Flat-ui has controls for filtering and sorting data, but it doesn't maintain its own control state; JSONDetail passes props for the control state and an onChange callback, and uses use-query-params to track the state in the URL:

const [query, setQuery] = useQueryParams({
  tab: StringParam,
  stickyColumnName: StringParam,
  sort: StringParam,
  filters: StringParam,
});

You pass useQueryParams an object of serializers (the keys are used as param names), and you can pass a partial object to the setter if you want to set only some fields. (There's a nice use of type-level programming to give query / setQuery the appropriate types based on the serializers.)

There is a lot of cool stuff going on in flat-ui and I would like to read that code some time.

Styling with Tailwind

All these components are styled with Tailwind CSS, so there are lots of class names sprinkled all over the place. I found this very mysterious! Here's what the Tailwind docs have to say about it:

Now I know what you’re thinking, “this is an atrocity, what a horrible mess!” and you’re right, it’s kind of ugly. In fact it’s just about impossible to think this is a good idea the first time you see it [...]

I really appreciate honest documentation!

As far as I understand the pitch: the traditional way to do CSS is to write "semantic" classes that describe the style for a particular component in the app, then use the class name whenever that component appears. The Tailwind way is to use "utility" classes that describe particular CSS attributes, and string together the ones you need when you need them. This might cost some duplication (similar components might use a similar set of utility classes), but saves the pain of writing and maintaining a hairy CSS file.

I've used styled-components, which is a different way to avoid writing and maintaining a hairy CSS file, so this makes sense to me; but it's a whole other language to learn. It would be cool if you could hover a Tailwind class in VS Code and see what it actually means... oh look.

That's it!

Please email me with comments, criticisms, or corrections.