
When the AI hype took the world by storm many companies started adding AI-infused capabilities to their product for the sake of it. VC money poured into every incumbent mentioning AI in their pitch deck, chat interfaces were everywhere. At Doist we're taking a different approach to building high-quality products. Read on to learn how AI allowed us to solve a longstanding, real-world user problem in a much simpler yet incredibly powerful way.

# The Opportunity

[Todoist’s filters](https://todoist.com/help/articles/introduction-to-filters-V98wIH) allow our users to find information quickly and efficiently, it’s ideal for those with large projects filled with tons of tasks. A filter consists of keywords and symbols, for example, `today | overdue`, and will return a list of tasks matching the criteria. However, the syntax is not always easy to grasp. As our CEO, Amir, [once said](https://twitter.com/amix3k/status/1659477109562015745):

> [...] we have an incredibly powerful filtering engine, but it takes a Ph.D. in algebra to use it.
>
> -- <cite>Amir, CEO of Doist</cite>

The proof was in the fact that only a small percentage of users were adding filters.

# The Solution

We were aware of the recent development of LLMs and their capabilities, and wanted to do some exploratory work on this whilst still addressing a solution to the real existing problem with filters. Thus, Todoist’s filters were the feature we landed on, based on low interaction rates and the assumption that the syntax surrounding forming filter queries is complex for new users.

We learned quickly that given a set of training data, AI would more often than not correctly return the intended results. However, given the nature of these queries (keywords alongside symbols, Todoist specific syntax for referencing projects, labels etc), we had some exploring to do to assert feasibility.

Here’s an example of a filter query to put into perspective the context required to form one:

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/filter-assist/non-ai-screen.png"
        alt="Existing Todoist filter modal"
    />
    <figcaption style={{ textAlign: 'center' }}>Existing Todoist filter modal.</figcaption>
</figure>

The above filter is just an example of what we can do with this powerful engine. However, this syntax would not necessarily be easily achievable by new users off the bat.

We initially explored approaching this from a non-AI perspective, by using different buttons and chips to include/exclude project names, labels, priority etc. We found that this was not easily solvable, however, and so shelved the idea. It would have introduced a ton of complexity in our UI and still left unanswered questions around whether this would help create filters.

_How could we assist these users in leveraging filters more effectively?_

-   **Our solution:** Use AI to easily create powerful Todoist filters ✨
-   **How:** Allow users to enter a natural language request (i.e. show me all of my overdue tasks), and return a generated filter.

# Technical aspects

## Backend

### API

We created a separate backend service for this work, which worked nicely as it decoupled us from other production code, allowing faster experimentation. We had to choose between two options: a Python based or a JS/TS based implementation.

-   **Python**
    -   Would allow for an easier transfer of ownership and support to the backend team given their existing knowledge in the language
    -   Most machine learning libraries are written in Python, so we’d be at the forefront of all updates
-   **JS/TS**
    -   All widely adopted tools have wrappers in JavaScript (i.e. langchain or OpenAI)
    -   Would mean frontend devs can contribute easily

Given a Python backend would satisfy our requirements, we opted for this. The result: a lightweight backend API built on top of FastAPI and served with [Uvicorn](https://www.uvicorn.org/).

---

### Endpoints

Our main endpoint was `/generate_filter`. This was the first endpoint called in our Filter Assist flow, and was responsible for sending the user’s input to the backend and returning the generated model’s response as a JSON object.

```
/generate_filter

Params
{
  "description": "All tasks due today",
    "language": "en",
}

Response
{
  "response_id": "52ab99b3dace4b939fc167f18c89ba4b",
    "query": "today",
    "name": "Today's due tasks"
}

```

We also added an endpoint to allow users to rate the response (either useful or not useful). The `rating_type` refers to whether the user explicitly interacted with our rating component (`EXPLICIT`) or asked to restart the process (`IMPLICIT`). In the latter’s case, we assumed a non-useful response.

```
/submit_filter_feedback

Params
{
  "is_useful": true
  "rating_type": "EXPLICIT"
  "response_id": "52ab99b3dace4b939fc167f18c89ba4b"
}

Response
{
    "response_id": "52ab99b3dace4b939fc167f18c89ba4b",
    "query": "today",
    "name": "Today's due tasks"
}

```

---

### Database

LLMs are at times unpredictable, and being able to investigate the quality of generated filters in regards to accuracy (high fidelity to what was asked in the input), as well as performance, was important and made us store data. Storing this data also meant we had an opportunity to further fine-tune. Since we were working with a simple model, we opted for DynamoDB. Our database went onto storing:

-   User input
-   The model’s output
-   Feedback sent on the output via the UI (more on this below)
-   Meta data such as timestamps and response IDs

<figure>
    <img
        src="/posts/filter-assist/dynamodb.png"
        alt="Database containing prompts, generated results and feedback"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Database containing prompts, generated results and feedback.
    </figcaption>
</figure>

## Models

Our first task was to evaluate different AI models available to us. With several developers working on the initiative, we decided to split up and each explore options and their results within the context of our filters.
We initially discussed building our own custom model and hosting it locally. As discussions matured, this approach turned out to be very expensive both from a time and infrastructure viewpoint. We wanted to ship something impactful to users fast. We also felt it would be a better approach to explore existing, well-tested implementations under a robust and reliable infrastructure first to get a sense of what exactly we needed from an AI model. We envisaged a custom model would come with a lot of overhead we may not have necessarily needed for our use case.

### OpenAI, closed-source or third party models

[Huggingface](https://huggingface.co/) is a popular AI model hosting platform for the machine learning community, giving individuals the opportunity to consume and contribute to open-source models. The platform contains hundreds of models, each one fine-tuned depending on the use case (text to speech, text generation, image to 3D etc).

A few of us started running some queries against these models, namely [Dolly](https://huggingface.co/databricks/dolly-v2-12b). Some findings:

-   Answers were returned between 11-150 seconds
-   Complicated questions resulted in a an out of memory error
-   Loading time was very slow (some 15 minutes)

Next up, we tried out GTP-3.5-turbo against GPT-4. Given 27 prompts on each, here’s what we found:

-   **GPT-3.5-turbo** = 19/27 ~ 70%
-   **GPT-4** = 26/27 ~ 96%

These results were obtained by tweaking our training & rules sheet. The first run using our first draft of training data & rules resulted in a much lower score. But these results proved to be extremely useful in benchmarking both of these models.

**Winning model:** OpenAI’s GPT4 model.

Via our new backend, we queried OpenAI’s [chat completions endpoint](https://platform.openai.com/docs/guides/text-generation) and specified our chosen model, along with other options such as max tokens and temperature (governing randomness and creativity). Our use case required as stable results as possible, so went for a low temperature (0.0).

---

### Training & testing data

We carried out a technique called few-shot prompting, which involves augmenting our prompt with examples and expectations. We compiled a large sheet with different natural language prompts (i.e. show me all tasks due today with a priority of 1) and their expected outputs in query syntax.

We then compiled a separate set of data; testing data, to input into the model after the few-shot process to judge its accuracy.

Finally, we decided to increase the user’s experience by also training the model to output not just a filter query, but also a suggested title. A few examples showing case the final format of our training data sheets:

| prompt                                                             | completion                 | title                      |
| ------------------------------------------------------------------ | -------------------------- | -------------------------- |
| Tasks that are overdue or due today that are in the "Work" project | (today \| overdue) & #Work | Today & overdue work tasks |
| Tasks that are due in the next 7 days and are labelled @waiting    | 7 days & @waiting          | Waiting next 7 days        |

## Frontend

By this point we had all the necessary puzzle pieces and configurations in place to train and obtain results from the OpenAI model:

-   A FastAPI backend
-   Endpoints for clients to generate a filter
-   A model
-   Training & testing data
-   A database to store results

We now needed to wire up our backend to Todoist’s UI to see this magic in practice!

Todoist already has a modal from which users can create filters using queries, we decided to extend this form to include two approaches to creating a filter:

-   The existing way, by entering the query directly
-   An AI assisted way, by querying our API with plain english

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/filter-assist/ai-start-screen.png"
        alt="Todoist Filter Assist start screen"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist Filter Assist start screen.</figcaption>
</figure>

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/filter-assist/ai-result-screen.png"
        alt="Todoist Filter Assist result screen"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist Filter Assist result screen.</figcaption>
</figure>

---

### UI Refinements

As loading times differ between models, we wanted to ensure we had proper handling of such a situation if this ever was a problem. So we added a loading illustration as soon as the form was submitted to indicate a pause until our API returned the generated filter.

To further improve the experience, however, we also added a filter preview (this feature was added to the filter modal as a whole, whether used with AI or not, as we felt more experienced users might benefit from it). Sometimes the AI result was close enough, but still required some tweaks specific to the Todoist content, in such a case the preview was extremely useful.

Finally, to support the analysis of our model’s accuracy, we included a rating component allowing users to send feedback on the generated query. Additionally, we also allowed for them to restart the process (resulting in an implicit “no, not useful” rating), or edit the input slightly and re-try.

## Challenges

### Complying with the feature’s characteristics

No solution comes without its challenges, and this one was no different. There were several problems we had to solve to ensure that our model would not return any non-functional filters to users. Todoist leverages Filterist, our own inhouse filter library facilitating its creation and parsing according to the allowed syntax and queries. We had to ensure that anything not currently supported when used outside of AI was not returned by the filter.

An example of this would be returning an unsupported symbol. To mitigate this, we leveraged:

-   A rules file. This file would contain a list of valid keywords and symbols. When training our model on testing data, we would ensure it also took into account these rules
-   A sheet containing `NOT SUPPORTED` examples as outputs
-   Our Filterist library validation on generated responses before returning the query back to the client

By design, we explicitly limit a filter’s max character count to 1024 in Todoist. Anything over this count would also be deemed an invalid generated filter when using AI. We were torn here between returning everything up to the max count, or invalidating it altogether. To reduce confusion, we opted for the latter. We specified a `max_tokens` property when querying OpenAI to safeguard against this, but we also added a rule in our rules file as an extra layer of protection: `The filter query can't be longer than 1024 chars, if it is, output NOT SUPPORTED.`

---

### Rate limiting

To protect against high-volume usage, we have rate limits in place that consider a number of factors like the user and platform.

## Final outcome

To recap, we wanted to use AI to easily create powerful Todoist filters. Our solution involved providing users with a UI to send plain English requests to a custom build API, from which we would invoke a GPT model to return a filter. We also added a filter preview and functionalities to retry and edit the request, and rate the experience.

Here’s a short demo of the final outcome 🎉:

<figure style={{ textAlign: 'center' }}>
    <video controls>
        <source src="/posts/filter-assist/filter-assist-demo.mp4" type="video/mp4" />
        <p>
            We can't play the video in your browser.{' '}
            <a href="public/posts/filter-assist/filter-assist-demo.mp4">Watch it here.</a>
        </p>
    </video>
    <figcaption style={{ textAlign: 'center' }}>
        The full flow show casing the creation of a filter using Filter Assist.
    </figcaption>
</figure>

This feature is just one of many included in Todoist’s Pro subscription, [sign up](https://app.todoist.com/auth/signup) to make the most of Filter Assist and other powerful features today ✨.


Natalie Mclaren

How we leveraged AI to assist users when using our filter feature.

Filter Assist: AI-Generated Filters in Todoist


2023 ended with our annual tradition of donating to open source. This time, we donated $5,189 across 17 projects and maintainers.

At Doist we cannot imagine a world without open-source software and its communities. For us, this means a world where people from different countries and cultures come together to collaborate on challenging projects out of curiosity, passion, and the desire to make things better. Just like we do with our products.

The 2023 round of donations came at a critical moment in world history. The macroeconomic situation was dire, and we saw waves of layoffs across the tech industry and companies cutting budgets, leaving maintainers and projects with uncertainty or no funding at all. This has motivated us to change our donation strategy from one-time payments to a recurring monthly donation, allowing creators and maintainers to rely on a steady and predictable stream of income instead of a single big payment towards the end of the year.

We're committed to continuing to support open-source projects to ensure their longevity and health. This sustained support of the technologies that form the backbone of the products we ship to users is not only about financial aid; it's also about being part of a vibrant, lively community and giving back to the people who make it all possible.

Here's the breakdown of our donations:

-   $1,032 or $86/month to [Gwendal Roué](https://github.com/sponsors/groue) (GRDB.swift)
-   $336 or $28/month to [Detekt](https://github.com/sponsors/detekt)
-   $336 or $28/month to [Nelson Osacky](https://github.com/sponsors/runningcode) (Fladle)
-   $336 or $28/month to [Alex Saveau](https://github.com/sponsors/SUPERCILEX) (gradle-play-publisher)
-   $204 or $17/month to [Pallets](https://github.com/sponsors/pallets) (Werkzeug, Click)
-   $204 or $17/month to [SQLAlchemy](https://github.com/sponsors/sqlalchemy)
-   $204 or $17/month to [pytest](https://opencollective.com/pytest)
-   $204 or $17/month to [Sebastián Ramírez](https://github.com/sponsors/tiangolo) (FastAPI)
-   $204 or $17/month to [Hynek Schlawack](https://github.com/sponsors/hynek) (structlog)
-   $516 or $43/month to [Jason A. Donenfeld](https://github.com/sponsors/zx2c4) (Wireguard)
-   $516 or $43/month to [Software in the Public Interest – Debian](https://www.debian.org/donations#spi)
-   $168 or $14/month to [Ariakit](https://github.com/sponsors/ariakit)
-   $168 or $14/month to [Mock Service Worker](https://github.com/sponsors/mswjs)
-   $168 or $14/month to [Claudéric Demers](https://github.com/sponsors/clauderic) (dnd-kit)
-   $168 or $14/month to [Daishi Kato](https://github.com/sponsors/dai-shi) (zustand)
-   $168 or $14/month to [Colin McDonnell](https://github.com/sponsors/colinhacks) (zod)
-   $168 or $14/month to [Sasha Koss](https://github.com/sponsors/kossnocorp) (date-fns)

A heartfelt thank you to all creators and maintainers, without which most of what we build wouldn't be possible. It's a tough and sometimes thankless job, yet it sits beneath it all. Keep at it and see you for another round in 2024.


Henning Muszynski

Why and how we donated $5,189 across 17 projects and maintainers during 2024

Our donations to OSS during 2024


# Introduction

At Doist, we build products that help millions of people every week stay organized and collaborate with each other. A great deal of what the software does in these products is to synchronize data across devices in a fault-tolerant way, embracing an offline-first principle for frictionless collaboration.

In recent months, our systems have undergone a significant evolution to serve more users and enhance their experience when dealing with more data.

To achieve this and to help troubleshoot issues in production, we have developed a system that allows us to provide a number of select engineers with secure and timely access to production environments, without compromising on accountability.

In this blog post, we'll walk you through the journey of how we got there and how it works.


# How were we doing it before

In the early days of the company, we ran code on dedicated long-lived EC2 instances.
Some developers got used to this and came to rely on the option to “debug in production” by connecting over SSH to such an instance and running a Python/iPython console to inspect system state: the combination of production code behavior and the environment it runs in, including database, cache, and some other runtime dependencies' state.

When we migrated services to [AWS ECS][aws-ecs], developers lost this option: there's no big long-lived production server anymore, system components run inside ephemeral containers, which are destroyed once the process running inside them exits.

We explored [AWS ECS Exec][aws-ecs-exec] to address the "debug in production" use case. However, these methods required installing [AWS CLI][aws-cli] and managing user permissions via [AWS IAM][aws-iam]. To retain the convenience of simply opening an SSH session and running an iPython console for system introspection, we aimed to replicate this experience using modern components that align with our new container-based setup instead of working against it.

[aws-ecs]: https://aws.amazon.com/ecs/
[aws-ecs-exec]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html
[aws-cli]: https://aws.amazon.com/cli/
[aws-iam]: https://aws.amazon.com/iam/

# The new way

Our end goal was a system with a simple a web interface, where developers could ask for an automatically provisioned dedicated short-lived server, that had the same code and environment as the most recently deployed version of our service.
Such a server should be available over SSH, and have an audit trail of actions executed over each SSH session.

## Providing SSH access

We assessed several options to provide SSH access to our containers and ended up with two final contenders: [AWS Systems Manager Session Manager][aws-ssm] and [Tailscale SSH][tailscale-ssh].

Both provided the ability to control who can access it through policies (AWS IAM and Tailscale ACLs respectively), and both provided the ability to record the session for auditing purposes.

From the get-go, we set session recording as a hard requirement.
This means that all actions taken within the environment will be recorded, providing an audit trail and ensuring accountability.

We ultimately decided to go with Tailscale SSH, as we already relied on Tailscale to limit access to some internal resources, and Tailscale SSH didn't require extra setup on the client (compared to using SSH sessions over AWS Systems Manager Session Manager). Tailscale SSH also added a [session recording feature][tailscale-ssh-recording] recently, which was enough to convince us to follow that route after a POC.

[aws-ssm]: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html
[tailscale-ssh]: https://tailscale.com/kb/1193/tailscale-ssh/
[tailscale-ssh-recording]: https://tailscale.com/kb/1246/tailscale-ssh-session-recording/


## The ephemeral console container

Now that we picked Tailscale as our SSH connectivity provider, we needed to figure out what and how to run it inside our container: it should resemble production, be accessible via Tailscale SSH, yet prevent automatic execution of production code on startup.

The first experiment was to run `tailscaled`, Tailscale's daemon process, as container entrypoint: `tailscaled` binary run as pid 1 process, configured networking, and spawned shells as needed for connected SSH clients. This prototype worked well, but to give ourselves more freedom, we decided to use a custom entrypoint process, that will:

- take care of additional setup;
- run `tailscaled` with appropriate flags;
- work as a real init process, reaping abandoned processes to avoid accumulation of “zombie” entries in the process table (just in case).

We creatively named this binary “bootstrap”.

The process tree on the newly provisioned system looks like this:

```
bootstrap (root, pid=1)
 ∟ tailscaled (root)
```

When SSH connects, the process tree becomes like this:

```
bootstrap (root, pid=1)
 ∟ tailscaled (root)
   ∟ bash (user)
```

Our goal was to have ephemeral servers use the same container image as production.

We knew that we need to bring a few additional components there, but we didn't want to directly include the bootstrap and tailscaled binaries in the production image.

The first idea was to build a new container image based on the production image:

```dockerfile
FROM production
RUN # install tailscaled and bootstrap binaries
```

but we abandoned this idea early: we have multiple releases each day, and wanted to avoid extra work done by CI pipeline.

Luckily, there was another way to get extra binaries inside a container based on the production image, without modifying the image itself.

We run our code in AWS Fargate, the “serverless” fully-managed counterpart of AWS Elastic Container Service.
The smallest unit of compute resource on AWS Fargate is a task — an ephemeral virtual machine provisioned with container image.

An AWS Fargate task may have more than one container running within the same VM, the so-called “sidecars”, often used for auxiliary purposes such as shipping telemetry or logs to an external system.
We decided to leverage this mechanism to provide additional binaries for our setup.
Containers running within the same AWS Fargate task may have some parts of their filesystems made available to each other via [cross-container mountpoints][ecs-bind-mounts].
This allowed us to make binaries from our “bootstrap” container appear inside a container running from an unmodified production image.

In [AWS CloudFormation template terms][container-defs], the relation between our containers within a single Fargate task takes the following shape:

```yaml
ContainerDefinitions:
  - Name: bootstrap
    Image: !Sub "${Repository.RepositoryUri}:latest"
    Essential: false
    ReadonlyRootFilesystem: true
    Command: ["/bin/sh", "-c", "true"]
    MountPoints:
    - SourceVolume: runtime
        ContainerPath: /var/runtime
  - Name: backend
    Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${OtherRepo}:latest"
    Essential: true
    EntryPoint: ["/var/runtime/bootstrap"]
    User: "0:0"
    VolumesFrom:
    - SourceContainer: bootstrap
        ReadOnly: true
```

[container-defs]: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinition.html
[ecs-bind-mounts]: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/bind-mounts.html

From our “bootstrap” container, we share the `/var/runtime` directory, holding only 3 binaries: `tailscaled` (Tailscale server), `tailscale` (Tailscale client, required to configure the server), and `bootstrap`, doing additional configuration and providing some convenience functions (more on that below).

Notice how the bootstrap container is marked as “non-essential” in the Fargate task definition, which means that when its container terminates right away, the whole task still continues to run, and we have a side effect of its subdirectory mounted within another container.


## The bootstrap process

The entire Dockerfile defining our `bootstrap` image looks like this:

```dockerfile
FROM public.ecr.aws/docker/library/alpine:latest as tailscale
WORKDIR /app
ENV TSFILE=tailscale_1.48.2_amd64.tgz
RUN wget https://pkgs.tailscale.com/stable/${TSFILE} && \
  tar xzf ${TSFILE} --strip-components=1

FROM public.ecr.aws/docker/library/golang:alpine as builder
WORKDIR /app
ENV GOFLAGS="-ldflags=-w -trimpath" CGO_ENABLED=0
COPY go.mod go.sum ./
RUN go mod download
COPY main.go .
RUN go build -o bootstrap .

FROM public.ecr.aws/docker/library/alpine:latest
COPY --from=tailscale /app/tailscaled /var/runtime/tailscaled
COPY --from=tailscale /app/tailscale /var/runtime/tailscale
COPY --from=builder /app/bootstrap /var/runtime/bootstrap
VOLUME ["/var/runtime"]
```

The bootstrap process, running as container entrypoint, takes care of the following:

1. Create a subdirectory used by tailscaled for its state management.
2. Launch `tailscaled` in [userspace networking mode](https://tailscale.com/kb/1112/userspace-networking/) as a child process.
3. Configure `tailscaled` (server) by calling `tailscale` (client) with appropriate flags, this tells the server to join our Tailscale network (tailnet) with a specific authentication key, explicitly set hostname, and enable Tailscale SSH.
4. Do some additional convenience configuration, like writing a custom `/etc/motd` file, and a few other things.

## Ephemeral Console Dispatcher

The Ephemeral Dispatcher is the controller system that spawns ephemeral consoles as ECS Fargate tasks.

It consists of an almost stateless web server, hosting a single web form, which developers use to select a few options and click the submit button, which provisions a new ephemeral instance.

Server logic is quite straightforward and is mostly glue code bringing together external components:
- It works as a limited-scope [OAuth client to Tailscale API](https://tailscale.com/kb/1215/oauth-clients/), which allows it to request new pre-authorized Tailscale authorization keys.
- It uses the AWS SDK to call [ECS RunTask][ecs-runtask], passing additional data such as Tailscale authorization key and termination delay, in the “Overrides” parameter.
- It works as a tailnet-only service, serving the web form only via Tailscale.

[ecs-runtask]: https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html

## Console's lifetime

We highlight the ephemeral nature of the console servers for users of this service, as we don't want them to treat such servers as ordinary long-lived VMs that persist across reboots, like some people used to with EC2 instances.
So right from the start, we made it mandatory in the provisioning form to select for how long the developer plans to use such an ephemeral server; like “I want this instance for 8 hours”.
The expectation is that such an instance is automatically destroyed after it runs for this long, and we don't need to worry about “garbage collection” or track how long they were not used.

Initially, we implemented the task retirement feature by having a sidecar container with a single sleep command running inside: if such container is marked as “essential” in the ECS task definition, and the sleep command exits after blocking for a specified amount of time, the whole task is terminated.

Soon enough, we got a feature request that it would be nice to somehow extend the lifetime of such an ephemeral server: for example, you started to debug something on an instance provisioned for 1 hour, but you realize it takes longer and want to have it run for a few hours more before it self-destructs.

We then moved this “sleep” functionality directly into the `bootstrap` process, which now knows how long it is expected to run.

This unlocked a few other convenience features:

- Now that the `bootstrap` process knows when it's expected to terminate, it writes this deadline into the `/etc/motd` file, so that the developer sees this in the banner shown after they log in over SSH.

- The `bootstrap` process keeps a timer and sends two early warnings — 30 minutes and 5 minutes before — to each logged-in user, by directly writing into `/dev/pts/*` device files, mimicking what the Unix `wall` command does.

We also taught the `bootstrap` process to adjust this deadline on demand.
To do so, the `bootstrap` process acts as a server, listening on a Unix socket.
On its launch, it also creates a symlink to itself under a different name (“expire-in”) under a directory inside `$PATH`.
When the `bootstrap` binary is called by this alternative name, it executes an alternative code path, working as a client communicating with the main `bootstrap` process over a Unix socket.
In this mode, it implements a small command-line tool used to adjust the instance expiration deadline and can be called like `expire-in 5h` to set the deadline to 5 hours from now.

## Filesystem persistence

We soon figured out that even though the ephemeral nature fits most tasks, for some of them, it's convenient to have some place where one can persist some files and re-use them across runs — such as ad-hoc scripts.

To make the “working from SSH console” experience more fluid, we rejected the option to store such data in S3 and instead configured a network filesystem share.
Conveniently, AWS provides [EFS][efs] — a managed NFS service, which we now use and mount a shared filesystem on each instance under a predefined path.
We put a colorful notice in our `motd` banner, telling developers where they can store files they want to survive each instance's lifetime.

[efs]: https://aws.amazon.com/efs/

## Conclusion

By building an internal system that leverages modern components like AWS ECS and Tailscale, we've been able to recreate the convenience of traditional debugging methods while addressing the challenges of production environment troubleshooting. This system has been running smoothly for several months and has received positive feedback from the developers that have used it.

If you enjoy building systems like this or solving problems with low-maintenance, elegant solutions that help fellow developers be more productive and efficient, [join us](https://doist.com/jobs/).


Artyom Pervukhin

The journey of how we built an ephemeral consoles system to enable timely and secure access to production environments, without compromising on accountability.

AWS ECS-based Ephemeral consoles for production issue troubleshooting


# Booleans and B-tree Indexes: A Cautionary Tale

Boolean fields are commonplace in most databases. They even featured prominently
in my [last post, about computed boolean flags](https://doist.dev/posts/computed-booleans).
However, when it comes to indexing, they can be a bit tricky.
Specially if your database is MySQL.

## Indexing - In Theory

On a very high level, a database index allows you to speed up queries by reducing the
search space and allowing you to traverse it faster. The mechanics behind them
will vary from platform to platform and from index type to index type.

The most common index type for relational databases is a [B-tree](https://dev.mysql.com/doc/refman/8.0/en/index-btree-hash.html). Suppose you
have an `is_active` column in the table that tracks your `User` objects. Let's assume
that 80% of your users are active, having `is_active = True`. A regular B-tree index on that column would split your search space into
two parts - one containing 20% of the records and the other containing 80% of
the records:

<figure>
    <img
        src="/posts/is-mysql/search-space.png"
        alt="Search space split by a proper index showing 20% of the users on the 'false' part"
    />
    <figcaption style={{ textAlign: 'center' }}>
        With proper indexing and index usage, searching through inactive users will involve dealing
        only with the little red region, instead of the whole search space
    </figcaption>
</figure>

If you search for records with a `False` value, for example, the database will
traverse only 20% part of the index to find the records you are looking for.

## Indexing - In Practice

Now, B-trees are great, but the devil sometimes lies in the details. When we added
an index for a boolean column we saw _radically_ different performance between
queries similar to:

```sql
SELECT * FROM items WHERE is_deleted = true;
```

and

```sql
SELECT * FROM items WHERE is_deleted IS true;
```

You can see a benchmark of that query over random data below, on a test performed
by Thomas Jost, a fellow Doister who worked with me on this analysis:

| Run           | IS FALSE | = FALSE |         |
| :------------ | :------- | :------ | :------ |
| Run 1         | 3,383    | 1,472   |         |
| Run 2         | 3,637    | 1,349   |         |
| Run 3         | 3,277    | 1,394   |         |
| Run 4         | 3,086    | 1,378   |         |
| Run 5         | 3,108    | 1,395   |         |
| Run 6         | 3,066    | 1,451   |         |
| Run 7         | 3,020    | 1,456   |         |
| Run 8         | 3,148    | 1,556   |         |
| Run 9         | 3,227    | 1,574   |         |
| Run 10        | 3,156    | 1,563   |         |
| Average       | 3,2882   | 1,4588  | -55,64% |
| Median        | 3,1915   | 1,4535  | -54,46% |
| Std deviation | 0,2692   | 0,0821  |         |

From that data, we can infer that _something_ is definitely off with our query.
But what? Let's build a simple example table:

```sql
CREATE TABLE `items` (
    `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    `is_deleted` tinyint(1) NOT NULL DEFAULT '0',
    PRIMARY KEY (`id`),
    KEY `idx_is_deleted` (`is_deleted`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
```

and take a look at the relevant part of the `EXPLAIN` for the two queries:

**Query #1**

```sql
EXPLAIN SELECT * FROM items WHERE is_deleted IS FALSE;
```

| possible_keys | key            | rows | filtered | Extra                    |
| ------------- | -------------- | ---- | -------- | ------------------------ |
|               | idx_is_deleted | 1    | 100      | Using where; Using index |

---

**Query #2**

```sql
EXPLAIN SELECT * FROM items WHERE is_deleted = FALSE;
```

| possible_keys  | key            | rows | filtered | Extra       |
| -------------- | -------------- | ---- | -------- | ----------- |
| idx_is_deleted | idx_is_deleted | 1    | 100      | Using index |

---

[View on DB Fiddle](https://www.db-fiddle.com/f/jTU6oeSm7kdbFGPKarqr1/0)

The second query uses the index, while the first one does not. Why is that? The
answer lies in the semantics of the `IS` operator.

## Identity vs Equality

In Computer Science it is sometimes important to distinguish between two types of
comparisons: **identity** and **equality**. The former is the strictest form of
comparison, where two objects are equal if and only if they **are** the same object.
The latter is a more relaxed check, where two objects are equal if they have the
same value.

This may seem like a trivial distinction, a splitting of hairs that is only relevant
for theoreticians and overzealous CS practitioners, but it is quite important.
If you ask me for _a_ beer, and I give you _a_ beer, we can say that the values are
equal.

However, if you ask me for _exactly the beer you bought yesterday_, and I give you _a_
beer from my fridge, the identity check fails, even if I give you a beer of
the exact same brand you bought.

In practice, the semantics of equalities and identities will vary between
different programming languages. If you ponder the complexities behind
the question "Is this object _the same_ as that object?", you can see why.

The difficulties of telling apart identity and equality are more pronounced
if you consider how easy it is for a program to copy parts of its own
memory bit by bit. If `x = 10` and `y = x`, are `x` and `y` pointing to the same memory address?
What if `a = 10` and `b = 10`? What if instead of dealing with trivial primitives like
integers, we are dealing with complex objects? The list of questions goes on and on.

## What is a Boolean for MySQL

In short, they don't exist. MySQL does not have a native boolean type.

Instead, it uses the `TINYINT` type to represent boolean values. As a `TINYINT` may range
from -128 to 127, MySQL will consider any value that is not 0 as true, and 0 as
false.

As one of our engineers, Artyom Pervukhin, pointed out, the nuance here is that
a query like `is_deleted IS True` crosses two different worlds - the
numeric realm where `is_deleted` lives and the boolean realm where `True` lives.

While `IS` and `=` will behave the same for values 0 and 1, they will differ for
any other:

```sql
MySQL [todoist_data]> select 0 is true, 0 = true;
+-----------+----------+
| 0 is true | 0 = true |
+-----------+----------+
|         0 |        0 |
+-----------+----------+
1 row in set (0.00 sec)

MySQL [todoist_data]> select 1 is true, 1 = true;
+-----------+----------+
| 1 is true | 1 = true |
+-----------+----------+
|         1 |        1 |
+-----------+----------+
1 row in set (0.00 sec)

MySQL [todoist_data]> select 2 is true, 2 = true;
+-----------+----------+
| 2 is true | 2 = true |
+-----------+----------+
|         1 |        0 |    <====== Aha!
+-----------+----------+
1 row in set (0.00 sec)
```

MySQL's documentation [states](https://dev.mysql.com/doc/refman/8.2/en/mysql-indexes.html):

> Comparison of dissimilar columns (comparing a string column to a temporal or
> numeric column, for example) may prevent use of indexes if values cannot be
> compared directly without conversion.

That is, this crossing between realms will prevent the use of the index.

Besides that, there is a nuanced difference when using expressions like `IS TRUE`
or `IS FALSE`. While it may seem that `IS TRUE` is directly comparing against
the `TRUE` as an alias for `1`, that's not the complete picture.
MySQL provides [three flavors for the `IS` operator](https://dev.mysql.com/doc/refman/8.0/en/comparison-operators.html#operator_is):

- `IS TRUE`
- `IS FALSE`
- `IS UNKNOWN`

When you use `column_name = TRUE`, you're applying the equals (`=`) operator
between the column value and the literal `1`. In contrast, `column_name IS TRUE`
applies the `IS TRUE` operator, where `TRUE` is _not_ a standalone literal
value but part of the syntactic operator.

## Conclusion

The conclusion is simple - **MySQL will not engage the index if you use `IS` instead
of `=`**. If you have boolean columns in your codebase and they are indexed, make
sure that your queries and the ones made by your ORM are using the appropriate operator.

The screenshot below showcases the number of slow queries before and after
we applied a system-wide fix, changing all `IS` operators to `=`:

<figure>
    <img
        src="/posts/is-mysql/impact.png"
        alt="Number of slow queries with booleans before and after the fix"
    />
    <figcaption style={{ textAlign: 'center' }}>
        The number of slow queries with booleans before and after the fix
    </figcaption>
</figure>

This small fix alone was enough to eliminate some performance issues
we were seeing under specific loads.

While surprising, this behavior is a [known issue](https://bugs.mysql.com/bug.php?id=95058).
Despite verified by the MySQL team, it is unclear if a fix will ever be implemented.

Lastly, it is worth noting that our tests further showed that:

- B-tree indexes _are_ engaged in comparisons using `IS NULL` in MySQL
- Maria DB is also affected by this issue
- PostgreSQL is not (see [DB Fiddle](https://www.db-fiddle.com/f/djrzhRynu5x4ehxYoqhiyC/0))

## Acknowledgments

I would like to thank Thomas Jost, Artyom Pervukhin, Alex Rodrigues, and Gonçalo Silva
for their help in researching and fixing this issue, as well as for their feedback
on this post.


Felipe "Bidu" Rodrigues

Are you sure your indexes are being used?

Booleans and B-tree Indexes: A Cautionary Tale


# Introduction

When it comes to ensuring quick and reliable delivery in the fast-paced world of software development, most organizations rely on gut feelings and manual pulse checks with their engineering teams. Others might follow inefficient or borderline harmful metrics like [lines of code written](https://waydev.co/lines-of-code-per-day/) to gauge the productivity of their teams. None of them are objective nor easily correlated with business results.

Luckily, there’s research on what to do instead – most famously the [DORA (DevOps Research and Assessment) metrics](https://dora.dev/). DORA metrics capture important aspects of the software delivery process, and by tracking them, provide a deeper understanding of operational performance, identify areas for improvement, and work towards consistently delivering highest-quality software at a high velocity.

In this blog post, we'll take a look at each DORA metric, discuss how we're tracking them at Doist, and share some insights and learnings from our experiences.

## Understanding DORA Metrics

DORA metrics were first introduced in the [2014 State of DevOps report](https://services.google.com/fh/files/misc/state-of-devops-2014.pdf) and later gained industry traction with the book [“Accelerate: The Science of Lean Software and DevOps”](https://www.goodreads.com/book/show/39080433-accelerate) by the same authors, [Nicole Forsgren](https://www.goodreads.com/author/show/17037914.Nicole_Forsgren), [Jez Humble](https://www.goodreads.com/author/show/4149510.Jez_Humble), and [Gene Kim](https://www.goodreads.com/author/show/328437.Gene_Kim). They are a set of four key indicators that help businesses measure their engineering performance.

1.  **Deployment Frequency** measures how often code is deployed to production.
2.  **Lead Time for Changes** tracks the time from code commit to code successfully running in production.
3.  **Time to Restore Service** quantifies how long it takes for a team to recover from a failure in the production environment.
4.  **Change Failure Rate** gauges the percentage of deployments causing a failure in production.

<figure>
    <img
        src="/posts/decoding-dora-metrics-at-doist/dora-report-2022.png"
        alt="Overview DORA Clusters from the state of DevOps report 2022"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Overview DORA Clusters, 
        <a
            href="https://services.google.com/fh/files/misc/2022_state_of_devops_report.pdf"
            rel="noopener noreferrer"
            target="_blank"
        >
            State of DevOps report 2022
        </a>
    </figcaption>
</figure>

There’s a fifth metric since 2022, **Reliability**, representing operational performance. It tracks how well services meet user expectations, such as availability and performance. For this article, we’re going to focus on the original four metrics only.

According to research, there are little to no benefits in optimizing your performance once you’ve joined the high performing cluster. Optimization for optimization’s sake is a waste of time. Time that’s better spent talking to engineers and leaders in your organization, uncovering and resolving actual problems they’re facing.

It’s paramount to keep in mind that DORA metrics are a conversation starter, not a solution. They lack deeper context on why certain metrics are the way they are in each organization. Hence, they cannot replace getting together with your teams to discuss processes and workflows and understanding their unique problems and opportunities.

# DORA at Doist

## Tooling

We have been pragmatic about our approach from day 1. Without knowing the benefits we would get from a more profound understanding of where we stood within each DORA metric, we weren’t convinced to invest in a dedicated observability tool. We’re tracking all metrics as [“Metrics” in Datadog](https://docs.datadoghq.com/metrics/), our main monitoring tool. We had experience piping data into Datadog and visualizing it there, so we decided to build upon this existing knowledge.

Most of the data collection is happening in our CI/CD setup on GitHub Actions. We’re leveraging the [datadog-actions-metrics](https://github.com/int128/datadog-actions-metrics) action to do most of the heavy lifting. The added benefit of this approach, is that we were able to build basic CI observability dashboards (how many actions ran, most costly actions, most flaky actions, …) from the same data – win-win.

## Doist's Definitions

### Deployment Frequency

Each engineering team measures how often their deployment pipeline (GitHub Actions workflow) runs and sends an event to Datadog upon successful completion. This works both for our Backend and Frontend teams, which deploy multiple times per day, as well as our Android and Apple teams, which usually deploy only once or twice per week.

### Lead Time for Changes

Every engineering team measures for how long each merged PR has been open for. We measure in seconds and report it in hours.

For now, we’re not overthinking it and treat all PRs as equal. Future iterations could consider the time a PR has been in draft state and exclude it from this metric. They may also exclude automated PRs. We'll see when we need to refine, but not before – Pragmatism beats perfectionism here again.

### Time to Restore Service

To measure “time to restore service” one would normally look at system metrics like error rate or availability and measure how long they take to recover. However, not all of our clients track this comparably.

We decided to define “time to restore service” as the time it takes to close a critical or high severity issue. This is less clear-cut as it takes time for a user to report an issue with customer experience, them reproducing and escalating it, and our engineers solving it. However, this is the flow we care most about optimizing, which makes this a healthy definition for our use case.

### Change Failure Rate

We define a change failure as a deployment leading to a critical or high severity issue being raised. Change failure rate is tracked over a 7-day rolling average, as not each deployment can be automatically linked to a reported issue.

Similar to “time to restore service”, this metric is less clear-cut as it’s not coupled to hard data like an error rate or real-user monitoring. Again, this makes sense in our use case to optimize the things we actually care about.

## Defining Failure

For both “Time to Restore Service” and “Change Failure Rate” we struggled to come up with a unified definition of **failure** across engineering teams. When we first set out to define and track DORA metrics, we had to accept a disconnect between the theory and our practice. In theory, we would want to characterize each new, unhandled error in production a failure of the system. Looking at our monitoring solutions (Sentry, Crashlytics, Datadog) this turned out to be unfeasible at the time.

As an example, our Frontend app reports 5-10 new errors per release. Some of them are valid and addressed swiftly (`undefined is not a function`, anyone?). Others are differently grouped occurrences of known issues. And still others might have causes outside our control.

As alluded to above, the right thing to do is clean it all up and always have a proper reporting & alerting hygiene in place. The reality of a bootstrapped product company is also that we need to keep working on features and fix bugs users are actually encountering (support tickets) as well as pay back ~10 years of technological evolution to eradicate whole categories of bugs instead of fixing them one by one.

We still want to get there, and eventually will, but we needed a more pragmatic approach to start. We settled on relating failure to support tickets, as closing them as quickly as possible is something every engineering team at Doist strives for. Measuring, reporting, and improving our handling of user-reported errors has the added benefit of being universally understood as a “good thing” by the entire company.

## Spilling the beans

After reading this far, let’s take a look at how Doist’s engineering organization is performing on these metrics:

<figure>
    <img
        src="/posts/decoding-dora-metrics-at-doist/dashboard-screenshot.png"
        alt="Screenshot of our DORA dashboard in Datadog"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Screenshot of our DORA dashboard in Datadog
    </figcaption>
</figure>

|                             | July       | August     | September  |
| --------------------------- | ---------- | ---------- | ---------- |
| **Deployment Frequency**    | 15 per day | 26 per day | 18 per day |
| **Lead Time for Changes**   | 63 hours   | 55 hours   | 101 hours  |
| **Time to Restore Service** | 177 days   | 174 days   | 32 days    |
| **Change Failure Rate**     | 3.38%      | 2.88%      | 5.53%      |

According to the 2022 State of DevOps report, this put us in the “high performing” cluster for all metrics but time to restore service, where we’re worse than their “low performing” cluster.

Entering the high performing cluster for time to restore service means being able to resolve most user reports in under a day. Technically, we have all the puzzle pieces in place to easily achieve this:

-   All our experimental features are behind feature flags, toggling a broken feature back off is a matter of minutes. No deployments required.
-   Rolling back a broken deployment or fixing it forward is similarly quick due to our continuous deployment setup. Pushing or reverting a commit is all it takes to get back into a healthy state.
-   While our globally distributed and fully async culture can cause some communication delays, it also brings a huge advantage of round the clock coverage for most of our teams. There will almost always be some online to address an issue.

Why are we looking at 32 days to restore service, then? There are at least two reasons for this:

1.  Coupling time to restore service to customer tickets is inherently slower than, e.g., automatic collection of crashes or other real-time service health metrics. There are too many hoops.
2.  By coupling this to our (rather large) backlog of customer support tickets, we skew the metric every time we close out an old issue. Or even worse, an old issue gets reopened just to be closed again.

Generally speaking, it seems likely we’re defining a “failure in the production environment” too strictly, and might fare better by coupling it to automatically reported error rates like the rest of the industry. Before changing our definition, we’re going to improve our testing, observability, alerting, and prioritization of issues to improve this metric into a single digit range. To join the high performing cluster of the next State of DevOps report, yes, but more importantly because it’s the right thing to do for users 💪

## Reflections

Our approach to tracking DORA metrics is simple, maybe even a tad too pragmatic. That’s because it is. We believe that **perfection is a journey, not a destination**, and that having some data – even if not perfect – is much better than having none at all.

Each month, we review our metrics, checking for and discussing any outliers. We value these discussions as they spark deeper insights, fueling our continuous improvement journey.

Our use of DORA metrics will be evolving in the future. Currently, they’re a mere pulse check for the health of our engineering teams and product delivery processes. However, as we continue to iterate, we’d like to make them the basis for informed staffing decisions. By coupling DORA metrics to SLOs we could use them to put more people on issue resolution or CD improvements if one of the metrics falls below a certain threshold. On the flip side, we can also confidently commit more people to feature work if our DORA metrics are healthy. By automating these decisions, we further streamline our processes and increase operational efficiency.

---

Lifting the veil of DORA metrics at Doist has revealed how we’re approaching this topic not only to improve our processes, but to foster a culture of continuous improvement and growth. We invite you to join us on this data-driven journey inspired by our relentless pursuit of excellence.


How DORA metrics can improve software delivery and offer in-depth insights into operational performance.

Decoding DORA Metrics at Doist


**TL;DR**: _In our tests, transitioning from a table that only has an `is_deleted`
field to a table that includes both a `deleted_at` field provided at write-time
and a computed `is_deleted` field stored by the database server results in
approximately an 8% increase in table size and a 2% decrease in `SELECT`
performance. We decided to make the switch due
to its positive effect on developer ergonomics and the intricate balance
between technical and human aspects of Software Engineering._

---

At Doist, our backend use soft deletes in some of our models to retain some historical data.
Similar to what is commonly done in situations like this, we have traditionally
used boolean flags, such as `is_deleted`, to denote whether a record is
considered deleted or not. That boolean-flag-based structure is useful
for any kind of scenario where a record's state needs to be stored in
the record itself, though it is not the only viable option for this.

Recently, we have evaluated using timestamp-based flags, such as `deleted_at` in
order to have a temporal marking of when a given record entered a state.
As interesting as this may be, swapping a pure boolean field for one
that handles dates and times is not without its cost. An alternative
to having only a timestamp such as `deleted_at` is to have both a
timestamp and a boolean field that is computed by the server
and used to speed up queries.

How it is now – we only know _if_ a record is or isn't deleted:

```python
>>> row = Row(data)
>>> row.is_deleted # False

Row.find(...) # by default, filters-out rows with is_deleted = True

def delete(row):
    row.is_deleted = True
    # Save and commit to the database
```

What we wanted – knowing if a record is or isn't deleted **and** when it was deleted:

```python
row = Row(data)
row.deleted_at # None, by default

Row.find(...) # Ignores rows with deleted_at != None

def delete(row):
    row.deleted_at = datetime.utcnow()
    # Save and commit
```

In this post, we'll discuss the benefits and drawbacks of each approach,
how we tested and evaluated them, and why we choose to move to timestamp
fields with stored booleans, even though they have a small performance hit.

## Boolean Flags & Date Time Records

Booleans are small pieces of data that only have two possible values: True and
False (or, jokingly, `FileNotFound`, according to
[this gem](https://thedailywtf.com/articles/What_Is_Truth_0x3f_) of Internet History).
Dates and times, however, have a wide range of possible values. If we consider
a resolution based on seconds alone, we have well over 30 million possible values.

This measurement of the possible size of a given set is known as the set's
cardinality, and it is an important concept when analyzing and predicting
database performance. Before doing any such migration, we were interested
in evaluating its impact on query performance. It is critical to note
that if a model supports soft-deletes, almost every query
will involve a filter like `WHERE NOT is_deleted`.

Touching a database column that is so commonly used around a system cannot
be taken lightly. If we have a performance hit in one of those areas,
it can quickly spread everywhere.

Apart from the sheer time that a given query occurs, another important
metric to keep in mind is the on-disk storage size of a given table and its indexes.
Datetimes are bigger than booleans by their own nature, so we
expected an overall increase in those metrics.

## Generate Columns

Broadly speaking, MySQL's
[generated columns](https://dev.mysql.com/doc/refman/8.0/en/create-table-generated-columns.html)
are calculated by the database server at some point in time – it can be at
insert or at select-time, for example. Those columns differ by
yet another case – if they are stored or not.

In MySQL parlance, we have the following nomenclature:

| Column Type | Computed At                  | Stored? |
| ----------- | ---------------------------- | ------- |
| `VIRTUAL`   | `SELECT`, when rows are read | No      |
| `STORED`    | `INSERT`/`UPDATE`, on writes | Yes     |

Stored computed columns are the ones we were most interested in. They would
allow us to, for example, calculate a boolean `is_deleted` based on whether
the `deleted_at` column is or isn't null. Having the server calculate
this means that we can let our developers not care about providing
values to this boolean column, but have it in place
and leverage it in queries.

### Experimental Setup

We tested four different table structures:

-   **Pure Boolean (Table A)**: A table with a simple `is_deleted` column
    that is indexed, similar to our current implementation.
-   **Pure Timestamps, not-indexed (Table B)**: A table with a `deleted_at` datetime column without an index
-   **Pure Timestamps, indexed (Table C)**: A table with a `deleted_at` datetime column with an index.
-   **Timestamps with computed booleans (Table D)**: A table with a `deleted_at`
    datetime column without an index and a **virtual, stored, boolean column is_deleted**
    that holds the value if `deleted_at` is or isn't null. This column is indexed.

We created these tables and populated them with varying numbers of rows
(1, 100, 1,000, 10,000, and 100,000) with randomized timestamps. We then
measured the time taken for insert operations and various select queries
using both Python-based tools and MySQL.

### Summary of Results

We can conclude:

-   Overall, the performance impact between pure booleans and computed booleans
    was small enough that we have a case to use computed columns to replace the current `is_deleted` field.
-   Pure booleans are more performant than datetimes or computed booleans, as expected.
-   For most analyses, though, there is no clear-cut difference between pure
    booleans, indexed date times, and computed booleans. Most differences
    are in the single-digit percentages.
-   Index size is the main factor that makes computed booleans more favorable
    than indexing date times. While indexed datetimes showed good performance,
    they also resulted in a bigger index size. This is expected,
    given the higher cardinality of datetimes.

## Insert Performance

Results from `%timeit` calls. Inserts were not batched to reflect better our regular usage, in which queries happen in isolation. This data represents a statistic based on 7 runs:

| Records | Boolean (Table A) | Timestamp, Not Indexed (Table B) | Timestamp, Indexed (Table C) | Timestamp Not Indexed + Computed Boolean (Table D) |
| ------- | :---------------: | :------------------------------: | :--------------------------: | :------------------------------------------------: |
| 100     |  96.8ms ± 1.16ms  |         96.1ms ± 1.05ms          |        109ms ± 2.86ms        |                   102ms ± 1.57ms                   |
| 1,000   |  1.05s ± 27.5ms   |          961ms ± 9.29ms          |       1.03s ± 51.3 ms        |                    1s ± 6.66ms                     |
| 10,000  |  10.5s ± 310.5ms  |         10.02s ± 82.3ms          |       9.94s± 490.2 ms        |                   11.5s ± 65.6ms                   |

We can see that inserting values to an indexed timestamp field is not significantly more costly than inserting data to a pure boolean field. The same can be said about computed booleans vs pure booleans.

While the data show small differences, the results are close enough that we cannot safely say that any one method is absolutely more performant than the other, while considering the timing aspects alone.

## Length Statistics

The tables were filled with a total of 500k random records each. The data below was reported by MySQL:

<figure>
    <img src="/posts/computed-booleans/data-size.png" alt="Table data size statistics" />
    <figcaption style={{ textAlign: 'center' }}>Data size statistics, from MySQL</figcaption>
</figure>

<details>
    <summary>Data statistics from MySQL</summary>

|                                           | `data_length` | `index_length` |
| ----------------------------------------- | :-----------: | :------------: |
| Pure Booleans (Table A)                   |  50,937,856   |   7,880,704    |
| Non-indexed Timestamps (Table B)          |  58,294,272   |       0        |
| Indexed Timestamps (Table C)              |  54,099,968   |   12,075,008   |
| Timestamps with Virtual Boolean (Table D) |  55,148,544   |   8,929,280    |

</details>

After inserting the records, some differences become clearer. As expected, indexed timestamps are the biggest indexes by far. As a matter of fact, the cardinality of the index for the timestamp field was `241,818`.

The table with a computed column (table D) had the second-biggest index, **about 13% bigger than standard booleans (table A)**. Besides that, the table with the computed columns was also bigger than the one without them (table C), which is expected as we are now keeping an additional value.

All in all, using the computed column increased the table size by ~8% when compared to pure booleans and by ~2% when compared with timestamps with indexes but without the computed column. **Overall, adding up data and index lengths, the combination of timestamp and computed boolean resulted in a table that is ~8% bigger than the one with just the boolean column (Table D vs Table A)**.

An interesting, parallel fact to note comes from comparing the table with indexed timestamps (table C) with the table with the computed boolean (table D). This shows that **using indexed timestamps creates a table that is about 3.2% bigger**. A finding showcasing the impact that a higher-cardinality index has on the overall size. The table with the computed column stores more data than the one with just the timestamp field and index, but it ends up being smaller.

## Select Performance

### Count — Python

All selects executed were `SELECT COUNT(*)` filtering for deleted rows. All tables had 500k rows. Timings are done with `%timeit`, results represent 7 runs:

<figure>
    <img
        src="/posts/computed-booleans/select-performance.png"
        alt="Table select performance statistics"
    />
    <figcaption style={{ textAlign: 'center' }}>Table select performance statistics</figcaption>
</figure>

<details>
    <summary>Select performance data</summary>

| Boolean (Table A) | Timestamp, Not Indexed (Table B) | Timestamp, Indexed (Table C) | Timestamp Not Indexed + Computed Boolean (Table D) |
| :---------------: | :------------------------------: | :--------------------------: | :------------------------------------------------: |
|  304 ms ± 683 µs  |         604 ms ± 1.54 ms         |       312 ms ± 844 µs        |                  310 ms ± 791 µs                   |

### Count — MySQL

| Boolean (Table A) | Timestamp, Not Indexed (Table B) | Timestamp, Indexed (Table C) | Timestamp Not Indexed + Computed Boolean (Table D) |
| :---------------: | :------------------------------: | :--------------------------: | :------------------------------------------------: |
|      325 ms       |              652 ms              |            329 ms            |                       333 ms                       |

</details>

Based on both datasets, pure boolean queries rank as the most performant, and queries on the indexed timestamp and computed boolean tables are close to a tie. **Considering the averaged timings, computed booleans (Table D) were ~2% slower than pure booleans (Table A)**

### Query Cost — MySQL

As reported by [`last_query_cost`](https://dev.mysql.com/doc/refman/8.0/en/server-status-variables.html#statvar_Last_query_cost):

| Boolean (Table A) | Timestamp, Not Indexed (Table B) | Timestamp, Indexed (Table C) | Timestamp Not Indexed + Computed Boolean (Table D) |
| :---------------: | :------------------------------: | :--------------------------: | :------------------------------------------------: |
|   45,198.119732   |          103,365.599000          |        47,476.020162         |                   47,309.717293                    |

Similar to select performance, pure boolean is a clear winner, and both indexed and computed booleans are very close together. **The difference between pure booleans (Table A) and computed booleans (Table D) is ~4%.**

It is worth noting that this analysis may be the least significant one, as query cost is more useful to compare different query plans for a similar query, and not between different queries altogether.

## DB Performance & Developer Experience

Besides performance, an important feature of any system that is intended to be maintained over a long period of time is **developer experience**. This matters very much in our case as Todoist is a codebase that is both a decade old and where we're just starting out.

**When software engineering-related aspects come into view, more subjective and less clean-cut aspects like maintainability and ergonomics around the code base become more relevant**. It is way harder to get a measurable metric for more human-related aspects of software development than it is to analyze technical benchmarks

From a purely technical perspective, making the change to computed booleans does not make much sense. But, from the human side of engineering, it does. Having a timestamp for when a given state was entered allows us to do more interesting and robust debugging and analysis.

**The fact that subjective metrics are harder to define and monitor makes them a prime target for under-prioritization**. This may not be a problem for prototypes and so on, but it is a problem when dealing with a codebase that we plan to keep sustainable, fun, and productive in the decades to come. At Doist, we proudly stand in the second group.

### Balancing Computed Fields vs. Manually Managed Fields

We _could_ have achieved our desired goal of using timestamps when dealing with a state flag and then converting it internally to a second boolean column or even just a boolean-like behavior by patching the code that accesses those data records. This would allow us to not have to deal with any kind of MySQL wizardry related to computing columns based on other columns and so on and so forth! So, why not?

The problem with that approach is that we would have to _centralize_ every access to our tables through some shared code that would do all of this handling. Sometimes we don't even _want_ or _can_ do that.

**Having the database server handling that computation gives us a guarantee that it does not actually matter how a record is written or updated, the relevant boolean flag will always be updated as well**, and we won't find ourselves in a situation where we have a record with a non-null `deleted_at` field with a false `is_deleted` flag.

### Testing Limitations and Pitfalls

Benchmarking systems is always a tricky endeavor. Several effects parallel to
the system under test can skew results and — even when the utmost care is
taken to minimize extraneous factors — these effects can still produce biased results.

In our particular case, we were not interested in making a broad, general claim
about the absolute performance of one approach or the other. Rather, we were
interested in their _relative, comparative performance_. For our intents
and purposes, putting all the approaches under the same experimental setup,
helps us get valuable data. We do recognize a few limitations of our approach:

-   Tests were executed on the same machine as the DB was running.
-   Tests were executed in a single-threaded manner with a single process.
    While this is useful for comparing strategies, it fundamentally differs
    from how most system works (single-threaded, multiple processes).

Please be aware of using any benchmark data to support claims that are different
than the ones that the benchmark was made to test! This can lead to bogus results.

## Bonus: Mixins over SQLAlchemy

With the data presented in the original research, our team set out to define it on an internal spec outlining our findings and how timestamp flags should be dealt with from now on. Our preferred choice of implementation was through mixins on SQLAlchemy. This allows us to easily add the more common soft-deleted columns in any models we like, and also provides us with a boilerplate code on how to implement similar timestamp-as-flag behavior in any other flag we feel like. Here is an example of that, implementing the "SoftDelete" mixin:

```python
from sqlalchemy import Boolean, Column, FetchedValue, func
from sqlalchemy.dialects.mysql import DATETIME
from sqlalchemy.orm import declarative_mixin



@declarative_mixin
class SoftDeletedMixin:
    deleted_at = Column(DATETIME(fsp=6), nullable=True, server_default=None)
    is_deleted = Column(
        Boolean, nullable=False, server_default="0", server_onupdate=FetchedValue()
    )

    @declared_attr
    def __table_args__(cls):
        return (Index(f"idx_is_deleted_{cls.__tablename__}", "is_deleted"),)
```

We hope that in publishing this article, we contribute back to the wonderful MySQL & SQLAlchemy ecosystems and provide a bit of insight on how we work with technical challenges in our daily tasks.


A comparison between regular boolean fields and MySQL's Computed Columns along the rationale behind changes in our codebase

Regular and Computed Booleans in MySQL


In his [excellent Android Dev Summit ’22 talk](https://www.youtube.com/watch?v=yJm5On5Gp4c) about Baseline Profiles, [Rahul Ravikumar](https://androiddev.social/@rahulrav) convinced me to adopt them and reap the benefits. In the process, we also set up [Macrobenchmark](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview) to measure the performance of our apps. For the most part, it was as easy to do as I thought, but throwing our Wear OS app into the mix added a few complications.

# What are Baseline Profiles?

A baseline profile is a file containing a list of classes and methods downloaded alongside your app when you install it from the Google Play Store. It is then used to compile those classes and methods ahead of time, resulting in startup, frame timing, and general performance improvements.

[Ahead-of-time compilation](https://en.wikipedia.org/wiki/Ahead-of-time_compilation) was introduced back in Android 5.0 (Lollipop), but before Baseline Profiles, there were always caveats with each strategy:

1. Complete ahead-of-time compilation took up too much installation time and disk space, since all methods and classes were compiled ahead of time.
2. [Profile Guided Optimization](https://source.android.com/docs/core/runtime/jit-compiler#flow) had poor first launch performance and was specific only to the installed device and app version (since Android 7.0 with [ART](https://www.youtube.com/watch?v=fwMM6g7wpQ8)).
3. [Play Cloud Profiles](https://android-developers.googleblog.com/2019/04/improving-app-performance-with-art.html) alone can take weeks to collect and generate an aggregated profile. It also requires profile regeneration with each app version (since Android 9.0).

The talk from Android Dev Summit ’22 mentioned earlier is a must-watch to learn about the history and trade-offs of these strategies!

## So what's new?

Well, with this new way of creating Baseline Profiles, these caveats are no longer applicable because now, you, the developer, can ship a profile with the app from day 1. We tried this for the Todoist app last month and the results certainly put a smile on our faces.

# The improvements

We have two Android apps for [Todoist](https://play.google.com/store/apps/details?id=com.todoist), the phone/tablet app and the Wear OS app. We measured the time to initial display (the startup time), duration of each frame (frame duration), and time by which a given frame missed its deadline (frame overrun time). Frame overrun time is available from API 31+ (Android 12), but the Google Pixel Watch runs Android 11, so we weren't able to measure it for the Wear OS app.

We used the very useful [Benchart](https://theapache64.github.io/benchart/) tool to convert the benchmark results into a much more readable format, and further verified our results using the Google Spreadsheet template from Py's blog post on [Statistically Rigorous Android Macrobenchmarks](https://py.hashnode.dev/statistically-rigorous-android-macrobenchmarks).

## Startup time

<figure style={{ textAlign: 'center' }}>
    <video controls style={{ width: '480px' }}>
        <source src="/posts/android-app-baseline-profiles/TD-Phone-Startup.mp4" type="video/mp4" />
        <p>
            We can't play the video in your browser.{' '}
            <a href="public/posts/android-app-baseline-profiles/TD-Phone-Startup.mp4">
                Watch it here.
            </a>
        </p>
    </video>
    <figcaption style={{ textAlign: 'center' }}>
        <strong>Left:</strong> Without Baseline Profiles, <strong>Right</strong>: With Baseline
        Profiles
    </figcaption>
</figure>

For the Todoist phone app, we saw a median improvement of about 23% in startup time, shaving off 328ms. The maximum startup time for the build with Baseline Profile was less than the minimum time of the build without it, ensuring a faster startup no matter what.

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/android-app-baseline-profiles/todoist-phone-startup.png"
        alt="Todoist phone app startup times"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist phone app startup times</figcaption>
</figure>

```
✅ min : After performed 20.6% better (-283.05ms)
✅ median : After performed 22.95% better (-328.05ms)
✅ max : After performed 27.94% better (-436.4ms)
```

<figure style={{ textAlign: 'center' }}>
    <video controls style={{ width: '480px' }}>
        <source src="/posts/android-app-baseline-profiles/TD-Wear-Startup.mp4" type="video/mp4" />
        <p>
            We can't play the video in your browser.{' '}
            <a href="public/posts/android-app-baseline-profiles/TD-Wear-Startup.mp4">
                Watch it here.
            </a>
        </p>
    </video>
    <figcaption style={{ textAlign: 'center' }}>
        <strong>Left:</strong> Without Baseline Profiles, <strong>Right</strong>: With Baseline
        Profiles
    </figcaption>
</figure>

At almost 14%, the improvement for the Wear OS app wasn't as drastic, but we were still able to save about 267 ms in the median startup time. Now, our worst startup time is better than the times most users were previously experiencing!

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/android-app-baseline-profiles/todoist-wear-startup.png"
        alt="Todoist Wear OS app startup times"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist Wear OS app startup times</figcaption>
</figure>

```
✅ min : After performed 19.96% better (-364.9ms)
✅ median : After performed 13.94% better (-267.7ms)
✅ max : After performed 8.5% better (-173.85ms)
```

## Frame duration

We saw some improvements in frame duration for the Todoist phone app. The longest 95th percentile frames improved by 19.96% (11.15ms) but otherwise the median duration was unnoticeably better.

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/android-app-baseline-profiles/todoist-phone-frame-duration.png"
        alt="Todoist phone app frame durations"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist phone app frame durations</figcaption>
</figure>

```
✅ P50 : After performed 3.38% better (-0.35ms)
✅ P90 : After performed 4.05% better (-1.2ms)
✅ P95 : After performed 19.96% better (-11.15ms)
✅ P99 : After performed 7.22% better (-12.3ms)
```

On the Wear OS app, we observed a similar level of improvement. The median frame duration only improved by 1.6%, but we saw a significant uptick in the extreme — frames took almost 35% less time in the 99th percentile.

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/android-app-baseline-profiles/todoist-wear-frame-duration.png"
        alt="Todoist Wear OS app frame durations"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist Wear OS app frame durations</figcaption>
</figure>

```
✅ P50 : After performed 1.61% better (-0.3ms)
❌ P90 : After performed 5.16% worse (+2.45ms)
✅ P95 : After performed 10.45% better (-10.7ms)
✅ P99 : After performed 34.62% better (-141.5ms)
```

## Frame overrun time

[Android documentation](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-metrics#frame-timing) describes frame overrun time as:

> How much time a given frame missed its deadline by. Positive numbers indicate a dropped frame and visible jank / stutter, negative numbers indicate how much faster than the deadline a frame was.

If this number is greater than 0, the overall smoothness of the app is affected negatively. Thankfully, in our case the median frame overrun time was already negative and didn't see much change (only 0.7ms) but even here, we saw improvements for the frames taking too long.

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/android-app-baseline-profiles/todoist-phone-frame-overrun.png"
        alt="Todoist phone app frame overrun times"
    />
    <figcaption style={{ textAlign: 'center' }}>Todoist phone app frame overrun times</figcaption>
</figure>

```
✅ P50 : After performed 2.08% better (-0.1ms)
✅ P90 : After performed 20.53% better (-3.85ms)
✅ P95 : After performed 45.86% better (-34.9ms)
✅ P99 : After performed 3.43% better (-9.05ms)
```

# Implementing it for Todoist and Wear

We have separate modules for the Todoist phone and Wear OS apps. After experimenting with having a single benchmark module for both, we ended up creating a dedicated module for each app to make it simpler to specify the `targetProjectPath` that Android Test modules require. It was also a better approach to use [Macrobenchmark instrumentation arguments](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-instrumentation-args) for generating a baseline profile from the terminal.

The official documentation to [set up a Macrobenchmark module](https://developer.android.com/topic/performance/benchmarking/macrobenchmark-overview#project-setup) and [create Baseline Profiles](https://developer.android.com/topic/performance/baselineprofiles/create-baselineprofile) came in very handy. Since we were already creating a new `benchmark` build type, it was easy to switch the network environment for these benchmarks to our staging server in that build type. This was very useful because, on the phone, we're going through the sign-up, onboarding, and new task user flows.

For structuring the tests, we used [the Robot pattern](https://academy.realm.io/posts/kau-jake-wharton-testing-robots/).

We also found that using `collectStableBaselineProfile` with a `maxIterations` of 7 for both apps yielded slightly better results for startup time than just using `collectBaselineProfile`.

Creating Baseline Profiles and measuring the performance on the phone app was straight-forward for the most part. The Wear OS app came with some challenges, though.

# Challenges

## Wear OS emulator on Apple Silicon

The phone app uses [Gradle Managed Devices](https://developer.android.com/studio/test/gradle-managed-devices) to generate the profile using a headless emulator. However, the Wear app needs a manually launched emulator, since there is [no support for Wear Gradle Managed Devices](https://issuetracker.google.com/issues/232753500) yet.

Yet another challenge with generating Baseline Profiles on a Wear OS emulator is that it doesn't work if the host machine is an M1 Mac. This is a [reported issue](https://issuetracker.google.com/issues/244619933). Our workaround was to use a separate x86 computer to host the Wear OS emulator, using a technique described in [How to use Android's x86 emulators on M1 Macs](https://doist.dev/posts/how-to-use-x86-emulators-on-m1-macs).

## Typing with UI Automator on Wear OS

Typing on Wear OS is different from the phone. When you want to prompt the user to type something, you send an Intent with `android.support.wearable.input.action.REMOTE_INPUT` action and the system shows these screens.

<figure>
    <img
        style={{ width: '360px' }}
        src="/posts/android-app-baseline-profiles/wear-inputs.png"
        alt="Wear OS input screens"
    />
    <figcaption style={{ textAlign: 'center' }}>Wear OS input screens</figcaption>
</figure>

So we need to find these Views without having their IDs, including the keyboard. It turns out that the content description of the keyboard button is `Keyboard`, of course. For typing itself, we have to find the `EditText` using a `By.clazz` selector, and finally to tap the "send" button, we need to use the `Send` content description. Here's what that looks like in code:

```kotlin
fun submitText(text: String) {
    val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
    device.findObject(By.descContains("Keyboard")).click()

    val editText = device.findObject(By.clazz("android.widget.EditText"))
    editText.text = text
    device.findObject(By.descContains("Send")).click()
}
```

## Running benchmarks on a Google Pixel watch

While the above issues were with the emulator, the benchmark was also very flaky on my Google Pixel watch, even when it was working perfectly on the emulator. Here are some ways to avoid that.

1. Enable the "Stay awake when charging" option in Developer Options.
2. Account for accidental double-taps. There might be times when a previous click goes through as a double-tap and UI Automator isn't able to find the element you're looking for anymore. Allowing the test to continue if some UI elements aren't found made it more resilient for us.

## Running benchmarks over Wi-Fi

Wireless debugging did not work very well for running the benchmark on the phone, due to connection drops, whereas _wired_ debugging worked perfectly.

## When to update Baseline Profiles

While work is ongoing to add support for CI, it isn't strictly necessary to generate Baseline Profiles very often. Since the profile contains a list of classes and methods, unless there's some refactor or new screens in the signup, onboarding, or the main content screen, there won't be a significant difference in performance.

For now, we have decided to generate them manually at every release, but only if necessary based on the changes. In the future, we plan to run it automatically in CI.

# Future

Baseline Profiles pushed us to start measuring our performance using the Macrobenchmark library. Now that we have that set up, we can work towards continuously monitoring the performance of our apps.

In the future, we'd like to set up an automated pipeline that stores the benchmark results from the last update, generates a new baseline, runs benchmarks, and compiles a comparison of the benchmark results between releases. There's a [sample](https://github.com/android/performance-samples/tree/main/MacrobenchmarkSample/ftl) from Google showing exactly how to do it using GitHub Actions and Firebase Test Lab, so it's just a matter of time.


Afzal Najam

How Baseline Profiles helped the Todoist phone and Wear OS apps get faster easily

Improving our Android app's startup time by 20%


Feature Flags are a powerful technique, allowing us to toggle features during testing, gradually roll out features without disrupting deployment, test new ideas on a subset of users, and fine-tune the user experience with A/B testing.

As of writing, Next.js lacks built-in support for Feature Flags, but they can be implemented in a myriad of ways, leveraging third-party tools. Our use case is to control page and feature visibility to A/B test ideas and understand how they perform. In this post, we’ll share our approach and experience implementing Feature Flags in Next.js using third-party libraries.

For our Next.js sites, we don’t use client-side navigation, e.g. [Next’s Link component](https://nextjs.org/docs/api-reference/next/link) and client-side hydration. There are reasons for this, including localisation and data fetching, but it meant we couldn’t maintain flag state between pages like you otherwise might. This difference also excluded access to the Web Storage API options, like local and session storage, as we need to check the flag state in Middleware and Next API routes.

Using Next.js’ `getServerSideProps` we could have initialised the flag service and evaluated the flags like in this contrived example:

```js
export async function getServerSideProps(ctx) {
    const flags = await FlagService.initialise({})
    if (!flags.betaPage.evaluate()) {
        return {
            notFound: true,
        }
    }
    return {}
}
```

This could work for just feature flags, but it wasn’t ideal for a few reasons:

-   The landing pages receive a lot of traffic, and we don’t want the service to become throttled or blocked
-   We’re unable to persist the A/B test information for the user
-   Next.js middleware would also be unaware of the user’s state

So we had some challenges, but we knew the goal. Minimise calls to the flag service while maintaining the flag state for the session and beyond with A/B tests.

## Session storage

Without the options for storing and maintaining the state outlined above, we started to investigate Next.js session storage libraries. [Next-Session](https://www.npmjs.com/package/next-session) and [Iron-Session](https://www.npmjs.com/package/iron-session) are two of the popular options that integrate well with Next.js.

We tested both options and decided that Iron-Session better fit our needs:

-   We didn’t need a Session Store like in Next-Session. Instead, Iron-Session would use a signed cookie,
-   The user couldn’t modify the flags; this is important for A/B tests,
-   It was well-documented, and
-   Supported Middleware and Next API routes out of the box.

## Iron-Session setup

With the choice made, here’s how we set up Iron-session in our Next.js sites.

First up is to install the dependencies so we can use them.

```bash
npm install -—save iron-session
```

If we update our contrived example from earlier, we can see how Iron-Session wraps `getServerSideProps`’s handler and introduces the session object on the request.

```ts
import { withIronSessionSsr } from 'iron-session/next'

export const getServerSideProps = withIronSessionSsr(
    async function getServerSideProps({ req }) {
        // session is now available
        const flags = req.session.flags

        if (!flags.betaPage) {
            return {
                notFound: true,
            }
        }

        return {}
    },
    {
        // iron-session configuration
    },
)
```

Let’s start taking this beyond a contrived example and into a working solution. We need to initialise the flag service, evaluate the flags and store them in the session, and they must be serialised because we are storing them on the cookie. Therefore, the flag needs to be evaluated before storage.

## Feature flag session

We decided to create a Feature Flag session function that would handle initialising our flag service, evaluating flags against the session and expiration (more on why that’s important in a bit).

We have abstracted the feature flag service out for the situation where we change providers. In that scenario, we need to support the `initFeatureFlags` call and `checkFlagState` and not worry about the specific details of each service.

Here’s what our Feature Flag session method looks like:

```ts
// Flag service is an abstraction for whatever flag service provider you're using to keep a consistent API
import { checkFlagState, initFeatureFlags } from './flag-service'

export async function featureFlagSession(
    session: IronSession,
    {
        evaluateFlags = [],
        evaluateABTests = [],
    }: { evaluateFlags?: FlagKeys[]; evaluateABTests?: ABTestKeys[] },
) {
    try {
        await initFeatureFlags()
    } catch (err) {
        // flag service failed to initialise
        // user will receive the default values
    }

    // Determine if the expiry has passed and if not return the flags from the store, otherwise an empty object
    const flagsExpired = session.flagExpiry ? session.flagExpiry < Date.now() : true
    const sessionFlagState: FlagSession = !flagsExpired && session.flags ? session.flags : {}
    const abTestFlags = session.abTestFlags ?? {}

    // If there are flags to evaluate, that aren't already in our store, do so against the service
    if (evaluateFlags) {
        for (const flag of evaluateFlags) {
            if (sessionFlagState[flag] === undefined) {
                sessionFlagState[flag] = checkFlagState(flag)
            }
        }
    }
    // If there are A/B tests to evaluate that aren't already in the store, do so against the flag service
    if (evaluateABTests) {
        for (const flag of evaluateABTests) {
            if (abTestFlags[flag] === undefined) {
                abTestFlags[flag] = checkFlagState(flag)
            }
        }
    }
    // Add the evaluated flags to the session
    session.flags = sessionFlagState
    session.abTestFlags = abTestFlags
    // If the flags have expired, set a new expiry date for one hour
    if (flagsExpired) {
        session.flagExpiry = Date.now() + 3600000 // 60,000 * 60 // 1 hour in ms
    }
}
```

Our flag session has the following params:

-   `evaluateFlags` - this accepts an array of flags to evaluate on the current route,
-   `evaluateABTests` - similar to the flags, we evaluate the AB tests currently running on the page

We could then use the Flag session within our `getServerSideProps` handler.

```ts
import { withIronSessionSsr } from 'iron-session/next'

export const getServerSideProps = withIronSessionSsr(
    async function getServerSideProps({ req }) {
        featureFlagSession(req.session)
        const flags = req.session.flags

        if (!flags.betaPage) {
            return {
                notFound: true,
            }
        }

        return {}
    },
    {
        // iron-session configuration
    },
)
```

## Expiry

You’ll notice that our Feature Flags have an expiry of 1 hour. We needed to do this because Feature Flags should be flexible, balancing the need to cache and update them. While once a user is assigned an A/B test group, they should remain in that group.

Iron-Session supports a single session store. While they can expire the cookie after some time, we couldn’t do that for the A/B tests because users would be reassigned to a different test group.

To work around that, we added an expiry to the flag session. When that time passes, we will ensure that we reevaluate the flags against the service.

We use an hour but will adjust this as we go and find an optimal value.

## Session wrappers

What this approach hasn’t solved is that we still need to handle the session in each route’s `getServerSideProps` handler. Including adding the iron-session configuration! This could definitely lead to bugs.

Ircon-Session has a recommendation for the setup called Session Wrappers, which are like handlers in Next.js for getServerSideProps and API Routes, and that’s what we will implement next to tie this all together.

```ts
export function withSessionSsr<P extends { [key: string]: unknown } = { [key: string]: unknown }>(
    handler: (
        context: GetServerSidePropsContext,
    ) => GetServerSidePropsResult<P> | Promise<GetServerSidePropsResult<P>>,
    {
        evaluateFlags,
        evaluateABTests,
    }: { evaluateFlags?: FlagKeys[]; evaluateABTests?: ABTestKeys[] } = {},
) {
    return withIronSessionSsr(
        async function (context) {
            await featureFlagSession(context.req.session, {
                evaluateFlags,
                evaluateABTests,
            })
            await context.req.session.save()
            return handler(context as GetServerSidePropsContext)
        },
        {
            // session options - this is where you'll have your passkey and other configuration for iron-session
        },
    )
}
```

The Session Wrapper will wrap the `getServerSideProps` handler like what `withIronSessionSsr` is doing now

```ts
export const getServerSideProps = withSessionSsr(
    async function getServerSideProps(ctx) {
        const featureFlags = featureFlags: context?.req?.session?.flags ?? {},
        if (!featureFlags?.betaPage) {
            return {
                notFound: true,
            }
        }

        return {
            props: {},
        }
    },
    {
        evaluateFlags: ['betaPage'],
    },
)
```

The SessionWrapper handles configuring Iron-Session, so we don’t have to each route and accepts params for evaluating flags and A/B tests and stores the data on the session by calling save. We can then access the flags on the session and continue using getServerSideProps as we do now.

We can add our feature flags to React Context if we want Client Side access via Next’s custom App. Most of the time, we found that validating within the data loading is sufficient.

Using a similar setup, we can also check the flags in Middleware and Next API Routes. There’s more information in Iron-Session’s documentation but here’s a Middleware example:

```ts
export const middleware = async (req: NextRequest) => {
  const res = NextResponse.next();
  const session = await getIronSession(req, res, {
    // iron-session config
  });

  const { flags } = session;

  if (!flags?.betaPage) {
    return NextResponse.redirect(new URL('/home', req.url))
  }

  return res;
};
```

## Wrapping up

And there we have it. We have feature flags within Next.js that we can access across pages, API Routes and Middleware without repeatedly calling a Flag Service. The approach worked well for our needs but wasn’t without tradeoffs.

As mentioned, if we are AB testing, we have locked them into a group. Typically this has been okay, but you lose some of the flexibility of flipping an AB test group all to true or false. We need to deploy this change. I am okay with being strict with A/B tests this way; we should use tests for a short period, make decisions, and move on, not leave them sitting around indefinitely.

There is a maximum size that a cookie can be, which is 4096 bytes. We have been okay with the small amount of information used in the session store, but it’s worth keeping that limit in mind if you start to go beyond a few feature flags.

Third and as discussed in the expiry section, having multiple session stores would have been nice. It would have simplified our approach between A/B tests and Feature Flags expiring by setting the expiry on the feature flag storage cookie instead. There are issues and discussions in the iron-session repo regarding this idea, and the maintainers explain well why it wasn’t done. We worked around it, as shown above, but it would still be nice.

This approach has worked well for Doist’s Feature Flags and A/B test requirements. We have had it in place for several months now. We use it frequently to test pages internally, limit access to certain site areas, and A/B test different marketing approaches to understand the most successful direction.

It would be great to see a native feature flag solution from the Next.js team, but until then, iron-session and a feature flag service are covering our needs nicely.


Anton Ball

Server side, Middleware and API route feature flags in Next.js with iron-session

Feature Flags in Next.js with iron-session


2022 is ending and with it, we have our [usual](https://doist.com/about-us#giving-back-section) round of donations to open-source software projects and maintainers. By sharing some details, I hope to inspire (read: pressure) other companies who can donate into doing so. After all, we're all building on top of this massive and wonderful infrastructure born out of a few people's skills, drive, and goodwill, that's mostly free to use. Humanity at its best.

Doist is no different. Open-source software is at our core. It's our first and largest technical building block, without which we couldn't have built any of our products. OSS has a [funding problem](https://stackoverflow.blog/2021/01/07/open-source-has-a-funding-problem/), and we try our best to be part of the solution, by sharing our gains, with projects we use, and recommending that others do the same.

This year, we had our customary budget for donations, inspired by the [1% for Open Source](https://www.onepercent.dev/) movement. Our engineering teams came together to select some of our most used and least funded OSS projects, allowing us to distribute the budget effectively and fairly across maintainers and projects relied upon by Doist. We also had a particularly slow hiring funnel for a role and leveraged some of its advertisement budget to publicly support a couple of projects related to it.

All things considered, we donated $7295 throughout 2022:

- $3500 to [Sebastián Ramírez](https://github.com/sponsors/tiangolo) (FastAPI)
- $500 to [SQLAlchemy](https://github.com/sponsors/sqlalchemy)
- $295 to [Ariakit](https://opencollective.com/ariakit)
- $295 to [Mock Service Worker](https://opencollective.com/mswjs)
- $294 to [dnd-kit](https://github.com/sponsors/clauderic)
- $221 to [Python Software Foundation](https://github.com/sponsors/python) (CPython, mypy)
- $221 to [Pallets](https://psfmember.org/civicrm/contribute/transact/?reset=1&id=20) (Werkzeug, Click)
- $221 to [uWSGI](https://unbit.it/uwsgi_donate)
- $221 to [Anthony Sottile](https://github.com/sponsors/asottile) (pre-commit, pytest, tox, flake8)
- $221 to [Gwendal Roué](https://github.com/sponsors/groue) (GRDB.swift)
- $221 to [Detekt](https://github.com/sponsors/detekt)
- $221 to [Nelson Osacky](https://github.com/sponsors/runningcode) (Fladle)
- $221 to [Jason A. Donenfeld](https://github.com/sponsors/zx2c4) (Wireguard)
- $74 to [Floating UI](https://opencollective.com/floating-ui)

A heartfelt thank you to all creators and maintainers, without which most of what we build wouldn't be possible. It's a tough and sometimes thankless job, yet it sits beneath it all. Keep at it and see you for another round in 2023.


Gonçalo Silva

Why and how we donated $7295 across 14 projects and maintainers in 2022

Our donations to OSS during 2022


# Async Engineering Mentorship

Asynchronous work is becoming mainstream, but that doesn’t mean it’s becoming easier. As beneficial as  is – and [oh boy it is!](https://async.twist.com/) –, it also comes with a few challenges which, worry not, are manageable and may even strengthen your overall process

When it comes to onboarding and mentoring new Engineers at Doist, we’ve been iterating on our processes over the years, making them as helpful and successful as possible. 

Below, you’ll find best practices and habits that helped us improve the success of our onboarding for new Engineers joining our team. 

**Bonus**: most tips are applicable to other fields.


## It starts before their first day!

Start thinking about your new team member’s onboarding from the moment you send them your offer.

The process should be thorough and tailored to their role, their profile, your team and, ultimately, your company.

This is important so that, when your new team member joins you, they have a solid and throughout support system behind them.


## Assign a dedicated mentor

Having a dedicated mentor to a newcomer means they have a stable person to reach out to, helping build a solid and personal connection in a (mostly) virtual environment from day one.

**Ideally, this person should be someone other than the team lead**. 

The team lead will have a different role in the person’s lifetime at the company and will have plenty of time to build a relationship together. 

Having a different person helps widen the newcomer's relationships and, as a nice side effect, empowers the mentor with a different role that provides more opportunities to grow in different skills like people management, work allocation, planning, and similar.


## Draft a plan

**Start by drafting a** **clear and** **intentional plan.** 

When working asynchronously, leaving things to chance or being too vague with expectations leaves others feeling blocked or lost, requiring  more feedback loops than necessary to find information and reflect on our progress.

If you're welcoming someone new to the team, this is especially bad. 

Instead, to help them succeed at their onboarding, draft a plan that’s tailored to their role and that includes all the information they need to succeed. 


## Set clear expectations 

Most interactions are text-based when working asynchronously. That makes it harder to convey emotions, or to “read the room”. 

If your newcomer doesn’t know what the expectations are, they’ll be in for a very stressful period which, compounded with the insecurity of starting a new job and trying to find their place in a new organization, makes it harder for them to succeed.

**If they know what they’re expected to achieve and what success looks like** in each part of their onboarding process, it’ll be much **easier to do the right thing** and, ultimately, enjoy their (hopefully!) dream job to the fullest.


## Provide feedback early and frequently

Ok, you’ve drafted a spectacular plan and your mentee knows what’s expected of them. Kudos! 

Now, **it’s also on your plate to be proactive and provide feedback frequently and clearly**. Whether it’s during 1:1s or asynchronously, do it and consider adapting your original plan based on what you observe and the feedback you receive.

If your mentee is rocking it and feeling the current plan is not ambitious enough for them, don’t be afraid to tweak it.

Or, on the other hand, if your mentee is struggling and needs to tone it down a bit, analyze it with them, by reflecting on what went wrong: were your expectations appropriate? Did the person have the right context and tools to excel? Are they going through a rough period and need to take it slower? 

Analyzing and adapting as needed is essential! 


## Onboard gradually

**Don’t bombard your new team member with information on their first day**. They’ll feel overwhelmed and most likely won’t absorb all the information they need.

Instead, pace yourself. **Expose them gradually and with a logical sequence** to different parts of your organization, codebase(s), processes and team rituals.

For example, when you hand out their first coding task, don’t ask for something that requires code review, handling translations, getting design assets, or that affects multiple components/modules in your codebase.

Start small, with a task that’s small and contained. That way, your new team member can check out the repo, get it up and running, perform the change they’re supposed to do, learn your approach to version management, and go through the usual peer review. Phew, that’s already a ton for someone who’s just getting started!

After they nail this, ask for something that requires interacting with design, or that requires translations, or some other new process, to expose them to a little additional workflow. 

They'll be comfortable with your routines and codebases in no time, and you didn’t even have to rush it!


## Promote autonomy from the start

Working asynchronously can be lonely and you’re never 100% sure when your colleagues will be available to help you. 

Not everyone is used to or comfortable with this aspect, so draft your onboarding process in a way that helps newcomers embrace this instead of fighting against it.

A few tips based on on how we do this at Doist:



* Enforce that it’s more than ok to explore the surroundings of your current task. They’re not wasting that time, they're investing it!
* Mention what’s useful to explore when they’re working on a given task: e.g., if you’re making a change on Todoist’s quick add, take your time to look around how input is validated, what edge cases we handle, how’s the interaction with the server, etc.
* Hold ourselves accountable to honoring a 24-hour turnaround time, as well as fully disconnecting during time off.
* Reassure that over-questioning and over-communicating is better than under-doing it.
* Make it explicit that they’re welcome to suggest ways to fill the gaps or a better way to do things.


## Async-first, not async-only

You’re working asynchronously _most_ of the time, not _all_ the time.

Schedule **regular 1:1s** both with the mentor and with the team lead, and encourage the person to schedule 1:1s with other people in the team for getting to know them.

If you can, consider **mentorship trips.** On these, the company sponsors travel and in-person time between the mentor and the mentee, allowing them to strengthen their relationship.

But don’t make it just about work 8h/day. Consider doing local activities, inviting team members that may live nearby, or simply spending time together.

---

Wrapping up, to successfully mentor engineers asynchronously, make sure to:



* Prepare ahead of time.
* Ensure they have all the resources they need to thrive - including time and a dedicated mentor!
* Do it gradually, and continuously adapt the plan to their progress.

We incorporate all suggestions above very deeply at Doist, which, compounded with [building newbie friendly codebases](https://doist.dev/posts/building-a-newbie-friendly-codebase), facilitates and streamlines our onboarding and mentoring processes.

Besides a higher-than-average employee retention rate at 88%, we’ve noticed a significant uptick in success and satisfaction with our onboarding process, with newcomers feeling thoroughly supported, and yet, treated as adults.

We hope these hard-earned lessons help you improve your processes and your newcomer’s success and satisfaction as well. Give them a try, and let me know how it goes!

🚀

Pedro Santos

Master the art of onboarding and mentoring Engineers asynchronously

Async Engineering Mentorship


Rasťo, our Head of Android, shared in [a prior post](https://doist.dev/posts/async-whiteboard-coding) how we came to leveraging an **async-first test project format** for our engineering roles in place of more traditional whiteboard coding.

The async test project helps us better understand candidate behavior in their day-to-day workloads, while leaving room for creativity and providing comparable input into the hiring process. It enriches other data points we gather through screening and structured interviews.

In today's post, we'll go deeper into how we put the approach Rasťo outlined into practice in one of the Doist teams, the Integrations team.

## The Shape

When candidates enter the test project stage, they receive an invitation to a private GitHub repository that only they and the interviewers can access.

The repository contains a small codebase as a jumping-off point for individual tasks. Giving candidates a [brownfield](<https://en.wikipedia.org/wiki/Brownfield_(software_development)>) project helps us better simulate real-world conditions and provides better comparability between candidates.

In it, they find the following items:

1. **README** – introduces the candidate to the repository, how to set it up locally, and the individual tasks to complete.
2. **Main branch** – contains a code sample forming the basis for all the tasks.
3. **Feature branch pull request** – contains a work-in-progress feature implementation for the candidate to review and provide feedback.
4. **Feature request issue** – A feature request for the candidate to implement as a pull request.
5. **Bug report issue** – A bug for the candidate to triage and resolve as a pull request.

<figure>
    <img
        src="/posts/async-test-projects-in-practice/image1.png"
        alt="A GitHub repository page with Issues and Pull Requests"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Notice the Issues and Pull Requests
    </figcaption>
</figure>

Once we invite candidates to their repository, we ask them to decide a deadline for delivering the test project. As an async company, we understand that our candidates have a life outside our interview project. Communicating about realistic timelines is more important than being able to work under time pressure.

## Async Preparedness

One of the key benefits of an async test project that happens wholly within GitHub is the ability to communicate with candidates as we would with our colleagues when discussing our work. Clear and concise communication is critical for async work; understanding the candidate's communication style is an explicit goal for our hiring process.

We encourage candidates to ask clarifying questions directly on the issues instead of debating through email. Likewise, we might comment on their feature with pull request comments or rebuke the feedback within their review.

One technique we discovered (accidentally) that helps us understand a candidate's problem resolution skills is to introduce a confusing element into one of the tasks. We did this by mistake when we gave candidates to fix a bug that didn't exist. Still, we learned so much from how candidates responded that it is, occasionally, helpful to introduce flaws into the tasks on purpose. 🤭

<figure>
    <img
        src="/posts/async-test-projects-in-practice/image3.png"
        alt="A GitHub Issue discussion rebuking an incorrect bug report"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Pawel rebuked a task to fix a bug
    </figcaption>
</figure>

## Automation

To rely on the approach above at scale, we automated the creation of the test project:

- **Generate a new repository** in a specified organization named after the candidate.
- **Push sample code** to the repository.
- Push feature branches to the repository.
- Generate **Pull requests and Issues** based on markdown templates.
- Automatically **invite the candidate** to the repository.

You can see my automation tool built with Node + TypeScript at [interviewing-repo-scaffolding](https://github.com/jankratochvilcz/interviewing-repo-scaffolding). After adding the requisite test tasks via Markdown annotated with [Front Matter](https://jekyllrb.com/docs/front-matter/), you only need to add the sample code and create the feature branch.

GitHub has [an excellent API](https://github.com/octokit/rest.js/), making most of the above steps a breeze. Generating repos, creating issues and PRs, and finally inviting the candidate is a call away.

For manipulating git to prepare local branches and pushing them to remote, we leverage [dugite](https://github.com/desktop/dugite), lightweight Git bindings for Node.

Once you've configured the tool locally, creating new test repos is as easy as inputting the candidate's GitHub username. 💪

Despite automating most of the process, there is more we'd like to improve. Most significantly, an integration with [Workable](https://www.workable.com/) or [Pinpoint](https://www.pinpointhq.com/) would enable us to automate the provisioning of test projects entirely when a candidate enters the test project stage, as well as cleaning up after the candidate concludes the hiring process.

## Learning Together

The mix of constraints, interactivity, and freedom that the test project format gives us has benefits beyond providing better assessment output.

As we model the test projects after real problems and codebases, it's common to learn about new approaches when reviewing candidates' test projects.

Candidates experienced with our tech stack sometimes show us neat tricks we can imitate. Novice candidates also occasionally point us to new tech stack features we're unaware of that can make us more efficient.

<figure>
    <img
        src="/posts/async-test-projects-in-practice/image2.png"
        alt="A GitHub Pull Request review with a suggestion to use a NestJS feature"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Pawel suggested a NestJS feature we were not aware of
    </figcaption>
</figure>

Conversely, we aim to provide constructive feedback on candidates' work via discussions within pull requests. We aspire to create mutual value even for candidates that don't end up progressing in our hiring process. It doesn't always happen, but when it does, the process becomes more engaging for both sides.

We hope that, if you haven't already, you will consider giving the test project format a shot and find it as helpful as it has been for us! 🚀


Jan Kratochvíl

An alternative test project format for your interviews

Async test projects in practice


Server-Driven UI has many interpretations, but in short, it boils down to an architecture where the Backend defines the user-interface (UI) of an application through its API responses.

On mobile, this term was made popular by Airbnb and Lyft,, who rewrote major parts and screens of their codebases to follow this pattern. To learn more about their experience, I highly recommend reading [“A Deep Dive into Airbnb’s Server-Driven UI System”](https://medium.com/airbnb-engineering/a-deep-dive-into-airbnbs-server-driven-ui-system-842244c5f5) by [Ryan Brooks](https://twitter.com/rbro112) and listening to the episode “[Server Driven UI with Kevin Fang and Jeff Hurray](https://podcasts.apple.com/us/podcast/server-driven-ui-with-kevin-fang-jeff-hurray/id1453587931?i=1000509742062)” from the Lyft Mobile Podcast.

As an example, here’s how we can map a JSON response into a UI Component using the Server-Driven UI SDK we built at Doist:

<figure>
    <picture>
        <img
            src="/posts/server-driven-ui-from-a-mobile-perspective/demo.png"
            alt="Server-Driven UI Example"
        />
    </picture>
    <figcaption>Figure: Server-Driven UI Example</figcaption>
</figure>


## Why Server-Driven UI?

Engineering is often based on trade-offs, and server-driven UI is no exception.

Advantages of adopting Server-Driven UI include:

* **Feature parity across all platforms via a standardized schema**, with well-defined boundaries and limits that each platform must support. These become the building blocks for building entire screens without any client-side changes.
* **Flexibility**, a whole screen can be changed on the server without the need for client-side changes, and therefore, releases.
* **Native UX**, since the server doesn’t send any HTML or web-based content. It sends a model that is mapped into native components by each client according to spec.

But there’s no balance without compromises. Some of the disadvantages we identified while integrating server-driven UI at Doist include:


* **It requires a lot of effort upfront**. If you implement a custom solution, you’ll need to define a schema and implement it in all clients.
* **Not as flexible as you might need**, after all, you’re using an architecture that promotes reusability across all platforms over small but often important details.
* **Potentially hard to maintain, **or as [Elye](https://medium.com/mobile-app-development-publication/the-possible-types-of-server-driven-ui-for-mobile-128e8c4433bb) beautifully explained:

> The success of Server Driven UI work within an organization cannot be determined by the initial release of the work. This is because the benefit of Server Driven UI can only be determined by how much change is required for future work

A lot more could be said about the advantages and disadvantages of server-driven UI. Each person and each company is using it in different contexts than ours with their own vision to it, and so, their experience will differ.


# How do we leverage Server-Driven UI at Doist?

At Doist, we have initiatives centered around integrations. With these, we aim to go where our core products can't and enable our users, partners, and ourselves, to leverage our core products in new and surprising ways. We want to allow third-party developers to easily integrate with our products, for example, to use [Toggl](https://toggl.com) with [Todoist](https://todoist.com), or [Giphy](https://giphy.com) with [Twist](http://twist.com/home).

Implementing each and every integration on each client wouldn’t be practical, and we couldn’t fathom having third-party developers depend on our team’s roadmap to thrive on our platform. This is where Server-Driven UI shines, as it allows us to uniformize our user experience across integrations and across platforms.

However, building a Server-Driven UI framework from scratch would take a lot of time and effort, and so we decided to look into existing solutions that would fit our needs without sacrificing flexibility.


## Adaptive Cards

We decided to go with [Adaptive Cards](https://adaptivecards.io), a Server-Driven UI framework by Microsoft. Their [JSON schema](https://adaptivecards.io/explorer/) felt appropriate to our needs and allowed us to skip creating one from scratch. It also had the scope for extensibility, so if there were any components that Adaptive Cards didn’t support, we were free to build our own and the Adaptive Cards SDK would accept those.

Microsoft has an editor called **Designer **that allows you to play with their Web SDK and test how any given JSON file would be rendered.

<figure>
    <picture>
        <img
            src="/posts/server-driven-ui-from-a-mobile-perspective/microsoft-designer.png"
            alt="Microsoft's Adaptive Cards Designer"
        />
    </picture>
    <figcaption>Figure: Microsoft's Adaptive Cards Designer</figcaption>
</figure>

We ended up building our own internal rendering SDKs leveraging the same schema, with a focus on:


* Using modern technologies to prepare us for the future as well as create a readable and maintainable codebase with **SwiftUI** and **Jetpack Compose**.
* Having solid test foundations with **unit** and **snapshot** **testing** to catch regressions and allow us to confidently iterate upon our codebases.
* **… a lot of sugar** to make our SDKs painless to use, with automatic image downloads and caching, vertical scrolling, and more!


## Was it worth it?

Building our own SDKs on top of the Adaptive Cards schema wasn’t a decision we took lightly. We looked deep into Microsoft’s SDK and tested it out before diving into the considerable task that is building an SDK from the ground up. It's still work in progress, but we’ve reached a point where we can build integrations, such as [Giphy](https://giphy.com) and [Jitsi](https://jitsi.org), and release them to our users.



<figure>
    <picture>
        <img
            src="/posts/server-driven-ui-from-a-mobile-perspective/twist-integrations.png"
            alt="Server-Driven UI in Twist"
        />
    </picture>
    <figcaption>Figure: Server-Driven UI in Twist</figcaption>
</figure>

You can try it **right now** on [Twist](http://twist.com/home), our async communciation app. There’s still a lot of work to be done, both in adding new supported elements as well as improving the overall user-interface and experience, but we’re happy with its evolution and looking forward to its future.



Pedro Carrasco

The what, why and how of Server-Driven UI at Doist from a Mobile perspective

Server-Driven UI from a Mobile Perspective


<figure>
    <picture>
        <source srcSet="/posts/synchronised-browser-debugging/hero.avif" type="image/avif" />
        <source srcSet="/posts/synchronised-browser-debugging/hero.webp" type="image/webp" />
        <img
            src="/posts/synchronised-browser-debugging/hero.jpg"
            width="2000"
            height="692"
            alt="iOS Simulator, Firefox, Android Emulator and Safari displaying Doist's landing page"
        />
    </picture>
</figure>

[Adam Argyle](https://twitter.com/argyleink) has a great series on YouTube called GUI Challenges about creating different UI components, breaking down the process, and challenging others to get involved. [It’s worth a watch](https://youtube.com/playlist?list=PLNYkxOF6rcIAaV1wwI9540OC_3XoIzMjQ) for any Frontend UI developers.

Something that stood out to me in the videos was Adam’s setup for browser testing. What he showcases in the affectionately titled _Debugging Corner_ chapter is something I wanted too.

<iframe
    width="560"
    height="315"
    src="https://www.youtube.com/embed/GDzzIlRhEzM?start=265"
    title="YouTube video player"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
    allowFullScreen
/>

The video gave a few ideas of how it’s setup:

-   Android Emulator,
-   iOS Simulator,
-   Regularly installed browsers, and the magic part
-   Browsersync keeping it all in sync

I wasn’t able to find a write up or video on Adam’s setup, so here’s how I, ahem, emulated the Debugging Corner.

## Browsers

The browser install was already handled. Safari is pre-installed on my machine and Firefox and Chrome were installed through Homebrew. But you can also head to the browser websites to install them yourself.

The Android Emulator, that took a little effort, so here’s how to get that up and running.

## Android Emulator

We are going to need Android’s command line tools to set up and run the Android Emulator and its requirements. There’s [packages available on the Android Studio site](https://developer.android.com/studio#command-tools) but it can also be handled with Brew.

```bash
brew install --cask android-commandlinetools
```

During the installation you may see this note about requiring Java. Make sure you have it available in some form to run the command line tools. I ended up using the recommended install command.

```bash
android-commandlinetools requires Java. You can install the latest version with:
brew install --cask temurin
```

Another output will be this message, which is important for later as we need to know where our Android SDK and tools are located

`Default Android SDK root is /usr/local/share/android-commandlinetools`

With the command line tools installed we can now set up the various SDKs, platform tools and system images, run:

```bash
sdkmanager "platforms;android-33" "platform-tools" "system-images;android-33;google_apis_playstore;x86_64"
```

I went with the most recent version of Android but this might be different when you are following these instructions. use the list command with sdk manager to see what’s available:

```bash
sdkmanager --list
```

This is handy in that it will also show you what’s already installed and let you manage updates. [Check the official documentation for more details](https://developer.android.com/studio/command-line/sdkmanager).

Next we need an Android Virtual Device, or AVD for short. This is what the emulator command will use to emulate the Android device. Think of it like the phone.

```bash
avdmanager create avd --name android-33-pixel --device pixel_5 --package "system-images;android-33;google_apis_playstore;x86_64"
```

There’s two required pieces of information. First, the name, which we will use to tell the emulator which device to emulate. The other is the package. This is what we installed with sdkmanager and is what describes the Android operating system.

I have also added a device profile, with pixel 5, but this part isn’t required. I just like to set it to a known device.

We are now ready to boot our Android device, remember that path from earlier? This is where we need that.

We can list the available AVDs the emulator can run with:

```bash
/usr/local/share/android-commandlinetools/emulator/emulator -list-avds
```

Then we can run the emulator by providing it the name of the AVD we want to use

```bash
/usr/local/share/android-commandlinetools/emulator/emulator -avd android-33-pixel
```

If you’re going to be running the emulator frequently it could pay to add it to your path or symlink it to your bin directory so you can run `emulator` instead of the full path every time.

There we go, we have a running Android device with Chrome ready for testing

<figure>
    <picture>
        <source
            srcSet="/posts/synchronised-browser-debugging/android_emulator.avif"
            type="image/avif"
        />
        <source
            srcSet="/posts/synchronised-browser-debugging/android_emulator.webp"
            type="image/webp"
        />
        <img
            src="/posts/synchronised-browser-debugging/android_emulator.png"
            width="512"
            height="1000"
            alt="Android Emulator running on Mac OS"
        />
    </picture>
    <figcaption>Figure: Android Emulator up and running</figcaption>
</figure>

## iPhone emulator

It’s a little easier to install the iOS Simulator.

Download Xcode from the App Store. When I did this it was 12gb! It’s a whopper of a program but with that comes an easier experience.

With Xcode installed, open the app and accept the terms and conditions. The simulators are now available.

I launched the Simulator app with Alfred but you can also find them in the Xcode app contents:

```bash
Applications/Xcode.app/Contents/Developer/Applications/Simulator.app
```

## Sync the browsers

The magic part of Adam’s setup is syncing the browsers across all devices, which is handled by [Browsersync](https://browsersync.io). It can be installed globally via NPM which is what I opted to do:

```bash
npm install -g browser-sync
```

We use [Next.js](https://nextjs.org/) for most of our projects, which has its own server, so we can use Browsersync’s proxy mode. If you have a different setup you might be able to integrate it differently, but this is fine for our needs.

In Terminal, change the current working directory to your Next project and kick off the server.

With another terminal tab in the same directory, run:

```bash
browser-sync start --proxy "http://localhost:3000/" --files "./"
```

Browsersync is now proxying to the NextJS server and watching for changes.

Open each browser and emulator you’d like in your Debugging Corner and navigate to the url that was output from running browsersync’s start command.

Now scroll. Magic! All the windows are in sync and showing across all the emulators and windows.

<video controls width="998">
    <source src="/posts/synchronised-browser-debugging/demo.mp4" type="video/mp4" />
</video>

## Next steps

I’m liking this setup both for testing and reviewing as I can test with greater coverage across browsers without repeating the same actions.

It would be great to automate more of the start. Currently I need to kick off each emulator and browser along with Browsersync and navigate to the required URL. Making that a single command would be nice, but I’m still seeing how this fits into my workflow.

Hopefully, you find this setup useful. Shoutout again to Adam Argyle for the idea and motivation.


Synchronised Browser Debugging


When I applied to join Doist as an Android developer eight years ago, I expected to do some whiteboard coding through a video call. That was, and in many companies still is, a common way to measure the coding proficiency of a candidate. It's also something that I dislike because being aware that someone is watching (and judging) me while I'm coding is very distracting.

To my surprise, after an initial interview, I was asked to do a take-home test project. The assignment was brief and practical, had a time limit of 10 hours, was compensated and I could do it in private and on my own schedule. These principles are all still part of our test projects.

Fun fact: A piece of my test project's code is still part of the Todoist codebase. This is something that’s always crossing my mind whenever I'm tempted to add a `// TODO(Rasto): Improve later` quick fix!

## Remote and async

In retrospect, a take-home test project was an obvious step for a remote and async company.

**It may well be that programmers good at whiteboard coding under supervision are also good at coding with an IDE in private. But the converse, in my experience, is not true.**

Therefore by applying this method we're discarding a subset of great candidates. Generally, it's done for two reasons:

- Convenience: It's practical and fast. The interviewer has a dedicated time slot for the interview and evaluation. And then they move on.
- Cheating: It's hard to cheat when someone's watching you, waiting for your response. In contrast, when the test project is done async, you have the whole Internet and (technically) more time available.

Both are true. They are also less relevant than picking the best candidates from the largest potential pool while creating a pleasant experience for the candidates. Regarding convenience, setting up a timeline for the candidate and the right process for all involved interviewers can minimise this very well.

As for cheating, using the Internet is perfectly fine. I'm using it daily when coding, why would I be bothered that a candidate does? Sticking to the expected time for implementing the test project is more important. It requires special care to choose the tasks that aren't significantly easier if you have more time. This is not a downside, because we don't want to evaluate candidates on how quickly they can type. Additionally, we have an interview with a developer only after the test project phase is successfully passed. One of the main topics there is to discuss the approaches taken in the test project and ask follow-up questions.

I'm also hopeful that whoever is competent to become an experienced software engineer realises that cheating at this step is not sustainable. When a 10-hour assignment takes you 20 hours to complete, how long are you able to endure working 16 hours a day once you are hired?

## Growing pains

As the years went by, Doist grew and became a more popular place to apply for. When we started expanding the Android team around 2018, we had a lot more applications than previously. We realised that to distinguish and evaluate the high-quality candidates we needed a more thorough test project. So we prepared one! It had a clear assignment and mockup, but we intentionally didn't make it explicit what, aside from the stated goal, we were expecting and assessing. The available time didn't allow the candidate to optimise for everything. The intention was to look for those candidates that valued what we value and focused on that.

Long story short, we failed to hire any candidate for over a year. I'm now painfully aware that we rejected multiple great candidates, simply because our expectations weren't explicit and we were critically judging any deviation from our model solution. In follow-up communication with some of the candidates, it turned out that some were even thinking of our desired approaches, but didn't take it for one reason or another. They simply prioritised something else than us and we didn't make that explicit.

Obviously, this would've been easier to notice during a live interview with two-way communication. But we're not doing those! For some time I was thinking about how to modify the assignment to make these expectations clear, yet the task not too easy.

## Day-to-day tasks

We set out to rework the test project with two main goals in mind:

- Clear and explicit objectives.
- Fast and objective evaluation.

The light-bulb moment came when we realised we didn't go far enough in reconsidering our implicit assumptions on what a test project should be like. Instead of focusing on creating a working project from scratch based on an assignment, we start with an existing project. And the assignment consists of several smaller and focused tasks that closely resemble the day-to-day work of our engineers. We're focusing on what we do the most, day in, day out.

One task is designed as a bug investigation based on a user report. Another task is a simple feature implementation based on a product request. Another task is doing a review of a pull request. This gives us a much broader overview of the candidate's skills and problem-solving approaches while saving time on trivial work, which comes with setting up a project from scratch.

<figure>
    <img
        src="/posts/remote-and-async-whiteboard-coding/example-task.png"
        alt="Example task from the test project README"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Example task from the test project README
    </figcaption>
</figure>

Complementary to this we created an evaluation sheet. Each task is reviewed and assigned one of four levels, based on defined criteria. Once all tasks are assessed, a threshold defines whether the project is passing and the candidate moves to the next phase. Reviewing the whole test project takes between 1 and 1.5 hours. This is about twice as fast as previously, saving also our engineers' time.

<figure>
    <img
        src="/posts/remote-and-async-whiteboard-coding/evaluation-sheet.jpeg"
        alt="Evaluation sheet template"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Evaluation sheet template
    </figcaption>
</figure>

## Room for creativity and feedback

The goal of the evaluation sheet is an objective assessment of the test project on its merit while minimising bias. At the same time, programming is a creative endeavour and the tasks are designed to focus on this aspect. This means candidates regularly come up with unexpected solutions that don't clearly fall into any level bucket. We had to keep this in mind and the evaluation sheet needs to be flexible to leave room for the reviewer's intuition and subjectivity.

Additionally, the test project assignment and repository aren't static and we need to keep them up-to-date. Partially, this is due to the Android platform and libraries evolving. We're always looking for feedback from candidates, both successful and rejected. Lastly, we're trying to spot patterns in the submissions to detect common unplanned pitfalls and address them.

## Conclusion

This setup started as an experiment on Android in late 2020. Since then, we've successfully used it in multiple hiring committees to hire four Android engineers. It's also being adopted across other teams at Doist. For me, the most pleasant consequence was getting positive feedback even from rejected candidates that they enjoyed the process and the assignments.

In the end, we're assessing candidates in how they perform on tasks that mirror what they'd be doing day in, day out. It feels good when things make sense.


Rastislav Vaško

Aligning our hiring with our day-to-day work

Async whiteboard coding



# Choosing a Multiplatform Stack

Developing two products — Todoist and Twist — for multiple platforms — Android, Apple platforms, Web, Windows, etc. — surfaces challenges in coordination and consistency. At Doist, we have been relying on common specifications to achieve consistency. Despite this, coordination is still hard and time-consuming, and for each specification update, no matter how big or small, we have to coordinate work for all platforms. Moreover, we generally lack shared tests to ensure implementations conform to their shared specification.

A multiplatform solution can solve both issues by concentrating specifications, tests, and implementation within a single software package. However, we care deeply about the great integrated experience that native platforms provide users, so end-to-end cross-platform solutions that bypass the native ecosystem such as React Native, Flutter, etc., are usually out of the equation for us. With these reservations in mind, what entices us most is **sharing common business logic**.

We are aware of challenges [[1](https://dropbox.tech/mobile/the-not-so-hidden-cost-of-sharing-code-between-ios-and-android)], but we have been testing for a while. A year ago, I revisited some of the multiplatform technologies with potential to fit our needs. In this post, I will share the process and the results.


## Assessment criteria and tested technologies

Looking for the best option to share business logic, the focus was on **performance** and  **interoperability** as the two major assessment metrics. There are other important aspects, but failing at any of these would be a serious blocker for us — we deeply care about fast and responsive interfaces, and a smooth and pleasant developing experience is no less of a concern.

I tested the following technologies:
- **JavaScript**, via iOS 14.3’s [JavaScriptCore](https://developer.apple.com/documentation/javascriptcore): common denominator for most host platforms and the underlying language for our Web clients.
- **Go**, v1.15: simple and easy to learn language that provides (limited) mobile support.
- **Rust**, v1.50: fast, safe, and modern language with a rich feature set.
- **Kotlin Multiplatform**, v1.4.21: modern language with very active development in becoming a multiplatform solution. We use it for Android, and could potentially adapt existing components and leverage existing internal knowledge and experience.

I left out C++ as the language could represent a critical first adoption barrier due to its complexity. On its replacement, I used Rust.


## Testing environment

Being part of the Apple team, I focused this exploration on the Apple platforms only. I used a small iOS app, in which I integrated simple libraries built on the mentioned technologies to assess their performance and interoperability with Swift. The app plots the time spent on the different tasks.

Those tasks cover multiple aspects that challenge both performance and interoperability from different perspectives. The following sections describe each of those tests, analyzing the results and reaching some conclusions.


### Hello test

This is the first typical “Hello, World” test. Each library exposes an API that accepts a `name` string parameter and returns the `“Hello, <name>!”` string as a result.

JavaScript implementation:

```javascript
function hello(name) {
    return "Hello, " + name + "!";
}
```

Despite how basic the test is, it already gives us important insights regarding performance and interoperability.

![Hello All](/posts/choosing-a-multiplatform-stack/hello-all.png)

As we can see, as the app calls the API 10k times for a list of randomly generated names, there are already some noticeable performance differences. Those differences are especially impacted by the interoperability between each library and the host iOS app. Particularly, on how the data (in this case, strings) is sent through the boundary.

I didn't investigate why Kotlin is significantly less performant than directly calling the native Swift API.

But I can explain why Rust, when invoked via [UniFFI](https://github.com/mozilla/uniffi-rs), is slower than when invoked via raw [FFI](https://doc.rust-lang.org/nomicon/ffi.html). UniFFI is a convenient library that provides safer and easier interoperability between Rust and other languages. In this case, Swift via an Objective-C interface. Behind the scenes, UniFFI is using FFI (i.e., a C interface), but to make it safer and simpler, it relies on [Lifting, Lowering and Serialization mechanisms](https://mozilla.github.io/uniffi-rs/internals/lifting_and_lowering.html) to send data across the boundary. This improves interoperability at the cost of performance degradation. As we can see in the previous graph, using raw FFI has a significant performance boost compared with UniFFI. However, interoperability is critically compromised! Dealing with the low-level C interface is not usable when dealing with slightly more complex data structures, such as arrays. Also, it is unsafe! Having two very safe and modern languages (Swift and Rust) communicating through such an unsafe layer (C) is, in my opinion, unreasonable.

So far, I can’t also explain why Go and JavaScript are that slower, but I believe that’s also related to sending data back and forth, i.e., interoperability issues. But we will investigate that in more detail in the following tests.


### Primes test

To verify the assumption that most of the performance issues are related to sending a high volume of data across the boundary, I added another test that returns the sum of all prime numbers up to the provided number. So, it only deals with native types (integers) for all the technologies (straightforward for crossing the Rust FFI layer, for example) but has a significant computational effort. I.e., near to zero performance cost in crossing the boundary.

Go implementation:
```go
func isPrime(number int) bool {
    if number < 2 {
        return false
    }

    for i := 2; i < number; i++ {
        if number % i == 0 {
            return false
        }
    }

    return true
}

func SumOfPrimes(until int) int {
    sum := 0

    for number := 2; number <= until; number++ {
        if isPrime(number) {
            sum += number
        }
    }

    return sum
}
```

Note: the algorithm is intentionally unoptimized, and it has the same implementation for all technologies.

![Primes All](/posts/choosing-a-multiplatform-stack/primes-all.png)

Surprisingly, Swift’s performance is terrible! Swift’s issue is related with the for loop that, even when optimized using [stride(from:to:by:)](https://developer.apple.com/documentation/swift/1641347-stride), has a poor runtime performance.

If you are curious, here’s the Swift implementation using `stride(from:to:by:`:
```swift
func isPrime(number: Int) -> Bool {
    if number < 2 {
        return false
    }

    for i in stride(from: 2, to: number, by: 1) {
        if number % i == 0 {
            return false
        }
    }

    return true
}

func sumOfPrimes(until: Int) -> Int {
    var sum = 0

    for number in stride(from: 2, through: until, by: 1) {
        if isPrime(number: number) {
            sum += number
        }
    }

    return sum
}
```

![Primes All but Swift](/posts/choosing-a-multiplatform-stack/primes-all-but-swift.png)

Analyzing all the other technologies clarifies the initial assumption. As the boundary is simple, and this test is not sending data back and forth, performance is more or less the same for both Kotlin and Rust (both UniFFI or FFI). The same happens regarding JavaScript, for which the algorithm itself does not perform that poorly, and there is no performance toll on crossing the boundary. I can’t explain why Go’s performance is worse, though. Like Swift, it’s probably not optimized for heavy computational tasks?


### Send items back test

To test the performance toll on crossing the boundary, I set up another basic test. It’s an API that accepts an array of simple data structures and returns it back without modifying it or doing anything else. The idea is to check the time spent on sending data back and forth only.

Rust implementation:
```rust
struct RustItem {
    name: String
}

fn rust_send_back(items: Vec<RustItem>) -> Vec<RustItem> {
    items
}
```

Note: I did not even test exposing this API using raw FFI, as sending arrays in both directions is a nightmare. I experimented with it, but complexity and safety compromises were too severe to consider this path. So, to use Rust in a practical way, we'd have to live with UniFFI or a similar high-level boundary layer.

At this point, we disqualified Go as an option, as [only a subset of Go types are currently supported](https://pkg.go.dev/golang.org/x/mobile/cmd/gobind#hdr-Type_restrictions), in which arrays, for example, are not included. Interoperability is still very compromised.

![Send Items Back](/posts/choosing-a-multiplatform-stack/send-items-back.png)

And the results are clarifying. Swift and Kotlin almost don't spend any time on this task as they are just sending back the same arrays, and both use native interfaces. But there's a significant performance cost on sending the arrays through Rust via the UniFFI layer, which is even worse than in the JavaScript case.

Focusing on Kotlin's interoperability with Swift (via an Objective-C interface) I tested sending Kotlin an array of a class defined in Swift that conforms to a Kotlin-defined interface. This way, we can decouple business logic in Kotlin that can be attached to existing codebases without requiring significant changes on the host platforms. In this case, the only requirement is to make Swift data structures conform to the Kotlin-defined interface.

Kotlin implementation:
```kotlin
interface KotlinItemInterface {
    val name: String
}

fun sendBackItemInterfaces(items: List<KotlinItemInterface>): List<KotlinItemInterface> {
    return items
}
```

Swift usage:
```swift
// Swift-defined data structure
class SwiftItemClass {
    let name: String

    init(name: String) {
        self.name = name
    }
}

// Conforming to the Kotlin interface
extension SwiftItemClass: SwiftItemInterface {}

// Sending Swift data structures to Kotlin
_ = kotlinImplementation.sendBackItemInterfaces(items: swiftItemClasses)
```

![Send Items Back via interfaces](/posts/choosing-a-multiplatform-stack/send-items-back-via-interfaces.png)

As we can see from the previous plot, such interoperability has no significant performance degradation.

### Map test

This test is a direct map function. I.e., items are mapped without any transformation.

Kotlin's implementation using interfaces:

```kotlin
fun mapItemInterfaces(items: List<KotlinItemInterface>): List<KotlinItemInterface> {
    return items.map { it }
}
```

![Map](/posts/choosing-a-multiplatform-stack/map.png)

Unsurprisingly, Rust via the UniFFI layer and JavaScript have significant performance degradation due to the boundary-crossing overhead.

So, we focused on Swift and Kotlin comparisons from now on, and on Kotlin’s interoperability possibilities, for example, to provide functions to perform a task in the host platform.

Kotlin implementation:
```kotlin
fun mapFunction(): (KotlinItemInterface) -> KotlinItemInterface {
    return {
        item: KotlinItemInterface -> item
    }
}
```

Swift usage:
```swift
func mapWithKotlinFunction(items: [KotlinItemInterface]) -> [KotlinItemInterface] {
    return items.map(KotlinImplementation().mapFunction())
}
```

![Map using functions](/posts/choosing-a-multiplatform-stack/map-using-functions.png)

Even though we are facing a slight performance degradation, it is good to know that we have multiple interoperability options using Kotlin Multiplatform via the Objective-C interface. Overall, Kotlin Multiplatform has been very promising on both performance and interoperability, the criterion we originally defined.

### Sorting test

This test is about returning the array of item objects sorted by their name. The results become interesting when using language-specific data structures, i.e., Swift is sorting Swift-defined structs, and Kotlin is sorting Kotlin-defined structs.

Swift implementation:
```swift
func sortSwiftItemStructs(items: [SwiftItemStruct]) -> [SwiftItemStruct] {
    return items.sorted { $0.name < $1.name }
}
```

Kotlin implementation:
```kotlin
fun sortKotlinItems(items: List<KotlinItem>): List<KotlinItem> {
    return items.sortedBy { it.name }
}
```

![Sort Swift vs Kotlin with structs](/posts/choosing-a-multiplatform-stack/sort-swift-vs-kotlin-structs.png)

I can't explain why Kotlin's performance is significantly better than Swift's, but I assume that Kotlin has faster sorting algorithms than Swift.

However, a more interesting approach regarding interoperability is to use interfaces.

![Sort Swift vs Kotlin via interfaces](/posts/choosing-a-multiplatform-stack/sort-swift-vs-kotlin-via-interfaces.png)

Again, there’s a slight trade-off between performance and interoperability.

Kotlin implementation:
```kotlin
fun sortKotlinItemInterfaces(items: List<KotlinItemInterface>): List<KotlinItemInterface> {
    return items.sortedBy { it.name }
}
```

Swift usage:
```swift
_ = kotlinImplementation.sortKotlinItemInterfaces(items: swiftItemClasses)
```

Note: we can’t send Swift structs because the boundary interface is in Objective-C. However, we are [looking forward](https://youtrack.jetbrains.com/issue/KT-49521) for direct interoperability with Swift for an even more powerful and seamless interoperability in the future.


## Conclusions

JavaScript is the native language for the Web and has almost ubiquitous support across platforms. However, its performance and interoperability characteristics are lacking. It shows a considerable performance degradation when crossing the boundary and on some processing tasks (sorting, for example). Interoperability is unsafe, and it requires significant boilerplate glue code (at least from the Swift side).

Go shows promise, but for our use cases, interoperability with Swift was very incomplete.I tried it most out of curiosity, as [Rust seems to be a more suitable choice](https://bitfieldconsulting.com/golang/rust-vs-go) for the specific use case of sharing business logic.

Rust could be a great fit. However, its interoperability with host platforms is problematic — FFI for Swift, JNI for Android, and WebAssembly for the Web. Simply looking at the Swift case, we’d have to make two modern and safe languages (Swift and Rust) communicate via an ancient and unsafe interface (C). There are alternatives, such as UniFFI, which provide safer interoperability (even though not as flexible and advanced as Kotlin’s) but at a significant performance cost.

Kotlin Multiplatform was the most promising technology in the tests and, in my opinion, the only technology worth exploring. It has good performance (for some tests, the generated binary is even faster than the native Swift implementations) and great interoperability. It’s still in Alpha, but JetBrains seems very active in developing it.


### Subsequent explorations

Since this exploration took place, we have continued exploring Kotlin Multiplatform. We have faced some interoperability challenges, especially on the Web, and I realized that some of my initial assumptions were wrong. For example, we found issues integrating the original architecture we designed for mobile clients in the Web platform. There, our architecture is primarily functional, so providing a list of data structures conforming to Kotlin-defined interfaces was not a good match.  Still, we were able to quickly [find another architectural solution that suited all host platforms](https://doist.dev/posts/filterist-kotlin-multiplatform-javascript-exploration).


Nuno Baldaia

Choosing a Multiplatform Stack


### Updates
- *Aug 24, 2022: Android Automotive emulator for API 32 is now available!*
- *July 13, 2022: Emulators for Android 5.x and 6.x (API 21-23) have also been released now!*
- *April 28, 2022: Android and Google TV emulator for API 31 and 33 are now available for Apple Silicon!*
- *April 4, 2022: Emulators for Android 7.x and 8.x (API 24-27) have been released!*
- *February 21, 2022: Wear OS 3 emulator for Apple Silicon is now available!*

---

So you bought that shiny new M1 Mac despite reading about [some](https://github.com/android/ndk/issues/1299) [drawbacks](https://issuetracker.google.com/issues?q=apple%20silicon%20m1%20status:open) for Android development and now you're wondering "how the heck am I gonna run all the emulators that aren't supported on this?" 

Well, you've come to the right place! But first, let's see what doesn't work and what to expect in the future. Here are the emulators that cannot run on the Apple silicon as of writing this:

- ~~All Android TV emulators.~~* (see update above)
- ~~All Wear OS emulators.~~* (see update above)
- ~~All Android Automotive emulators.~~* (see update above)
- ~~Android 8.1 (API 27) and below.~~* (see update above)

While [Android TV](https://issuetracker.google.com/issues/203238268) emulator images are eventually going to arrive this year, it is unlikely that API 23 and lower will ever be supported.

What if I told you that you can run them and **debug** from your new Mac today, granted you have another x64 computer lying around.

For the purpose of this article, we'll call the computer running the emulator, the host PC, and the computer running Android Studio, the guest PC. We will do it on a Windows host but it _should_ work the same way on a Linux or macOS host.

1. First, enable Remote Desktop on the host PC. It's in the Settings app on Windows 10 and 11.

    <figure>
        <img
            src="/posts/androids-x86-emulators-on-m1-macs/windows-remote-desktop-setting.png"
            alt="Remote Desktop setting on Windows 11"
        />
        <figcaption style={{ textAlign: 'center' }}>
            Remote Desktop setting on Windows 11
        </figcaption>
    </figure>

  If, like me, your host PC needs to be available for other people to use at the same time you need to run the emulator, check out the instructions [here on how to allow multiple sessions in Windows 10 and 11](http://woshub.com/how-to-allow-multiple-rdp-sessions-in-windows-10/).

2. Install Android Studio on both computers. It's not strictly necessary to do it on the host PC since you can create and run the emulator using command line, but we're just doing it for simplicity.

3. Download and create the emulator you want on the host PC. For example, I created an Android TV emulator on my Windows machine.

4. On the host computer, setup an SSH server. On Windows, an easy way is to install OpenSSH server using PowerShell.
    ```powershell
    Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
    ```

5. Start the service.
    ```powershell
    Start-Service sshd
    ```

    Make it run automatically on startup (optional).
    ```powershell
    Set-Service -Name sshd -StartupType 'Automatic'
    ```

6. Confirm that the firewall rule is set. It should be created automatically during setup. Run the following to verify:
    ```powershell
    if (!(Get-NetFirewallRule -Name "OpenSSH-Server-In-TCP" -ErrorAction SilentlyContinue | Select-Object Name, Enabled)) {
        Write-Output "Firewall Rule 'OpenSSH-Server-In-TCP' does not exist, creating it..."
        New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
    } else {
        Write-Output "Firewall rule 'OpenSSH-Server-In-TCP' has been created and exists."
    }
    ```

7. Run the emulator you want to use on the host PC now.

8. Use ssh to forward port 5555 on the host PC to the guest PC. To do this, run this command where `192.168.2.12` is the IP of the host PC:
    ```shell
    $ ssh -NL 5555:localhost:5555 android@192.168.2.12
    ```

9. Go ahead and connect ADB.
    ```shell
    $ adb connect localhost
    ```

10. All that's left now is to authorize the computer on the emulator when the dialog popups up.

    <figure>
        <img
            src="/posts/androids-x86-emulators-on-m1-macs/android-tv-emulator.png"
            alt="Android TV emulator showing a debug authorization dialog"
        />
        <figcaption style={{ textAlign: 'center' }}>
            Android TV emulator showing a debug authorization dialog
        </figcaption>
    </figure>


If you followed the steps, Android Studio on your guest PC should see the emulator on the host PC. This allows you to run and debug the apps as if the emulator was running locally, circumventing architecture-specific limitations you might face on your M1 Mac.

<br/>

<small>
    Cross-posted on <a href="https://afzal.me/how-to-use-androids-x86-emulators-on-m1-macs" target="_blank" rel="canonical">Hashnode</a>. Leave a comment over there!
</small>


Leveraging another PC for unsupported Android emulators

How to use Android's x86 emulators on M1 Macs

One of our engineering values at Doist is **Continuous Improvement**. We apply it in various situations, such as opportunistic refactoring, or performance improvements. Continuing [our efforts](https://doist.dev/posts/android-app-continuous-deployment) around **Continuous Deployment** (CD), in this post, I'll focus on CD for iOS to ship our apps faster and get continuous feedback from not only customers but also our team.

CD encourages that all the code you merge to the main branch is deployable at any time. We pair it with [feature flags](https://www.cloudbees.com/products/feature-management), to turn certain functionality on and off remotely without deploying code. Feature flags allow us to work in small batches. This enables small and atomic pull requests, making reviews easier, and reducing lead time.

At Doist, we have 9 iOS developers working on [Todoist](https://todoist.com) and [Twist](https://twist.com). As we're a relatively small team, we are improving our deployment process to help us focus on impactful things. Plus, CD on iOS requires a different approach as the release process is tightly connected with the App Store and App Store Review.

With this in mind, I'll share how we approached some of the challenges related to CD. Let’s dive in.

## Release cycle

First, let's talk about our release cycle. Uploading a release build is always triggered from the main branch and is automated as scheduled events. We mainly use [GitHub Actions](https://docs.github.com/en/actions) and [Fastlane](https://fastlane.tools) to automate our workflows.

In production, we release our apps **every week** on the same day of the week via the App Store. Having a fixed release cycle eliminates unnecessary communication regarding releases and fits our asynchronous approach to work.

Internally, we release the apps **every day** via TestFlight. We're limited to 100 internal testers as of today, but being able to start testing immediately without needing a review is a huge advantage. We also recommend that Doisters turn on Automatic Updates in the TestFlight app so that they can use the most up-to-date version as soon as it's available.

## Feature flags

We use feature flags to isolate code that we don't want to release in production yet. For example, the code may contain work in progress, untranslated content, or new features that we plan to release on a specific date. With that, the main branch is always ready to ship at any time, making the release cycle independent from the development process.

When it comes to new features, a new feature will be only available internally first. And then, in the case of Todoist, public beta users will see it if they turn on [Experimental Features](https://todoist.com/help/articles/experimental-features). Once we improve it based on feedback, we'll officially release it to everyone via feature flags.

For better internal testing, we provide a debug view to override feature flags locally.

<figure>
    <img
        src="/posts/continuous-deployment-for-ios/feature-flags-debug-view.png"
        alt="Feature flags debug view"
    />
</figure>

One of the pitfalls that we didn't initially pay attention to is a flag freeze. When a flag value is updated remotely, a flag value in the app needs to be synced. Flag freeze is to keep the flag value unchanged until future evaluation. It's needed to prevent unexpected states due to flags being dynamically updated while the app is running. In the worst case, the app will crash. Even though this crash can be seen as a one-time issue, we took it seriously as it can lead to data loss if a user is trying to save something.

One way to avoid that is to keep a flag value consistent until the next launch. This will result in consistent behavior within the same session, for both customers and our team.

Creating a remote feature flag doesn't come for free as we need to set it up and manage it on a dashboard. In case we need a local feature flag that we can enable in development, we have another mechanism called Debug Feature Flag based on the debug build configuration in our projects.

```swift
public struct DebugFeatureFlag {
    private let isEnabledInDebug: Bool

    public init(isEnabled: Bool) {
        isEnabledInDebug = isEnabled
    }

    public var isEnabled: Bool {
        #if DEBUG
        return isEnabledInDebug
        #else
        return false
        #endif
    }
}
```

This is useful for simple use cases, such as isolating untranslated content, or small features that we won't release soon. When it's ready, what we have to do is remove the flag locally.

In a dedicated file, this is how we use it.

```swift
if DebugFeatureFlag.myFeatureFlag.isEnabled {
    // Debug code
} else {
    // Production code
}

...

private extension DebugFeatureFlag {
    static let myFeatureFlag = DebugFeatureFlag(isEnabled: true)
}
```

Another common challenge after adopting feature flags is managing their ownership and expiration. We make it explicit in the code itself, following this format.

```swift
// Owner: [Name], When to remove: [Tentative Date]
let xxx = RoxFlag()
```

In addition to that, we use a monthly check on Twist.

<figure>
    <img
        src="/posts/continuous-deployment-for-ios/feature-flags-monthly-check.png"
        alt="Feature flags monthly check"
    />
    <figcaption style={{ textAlign: 'center' }}>
        A monthly reminder using Twist Team check-in Integration
    </figcaption>
</figure>

This is not the ideal workflow, but it works well without too much effort. We like starting small first to understand the root problem and trying to apply Continuous Improvement where possible.

## Versioning

Before introducing CD, we used [Semantic Versioning](https://semver.org) (`[MAJOR].[MINOR].[PATCH]`) like most apps in the App Store. At the same time, we struggled to decide when to bump a major or minor version.

Now that most of our new features are released via feature flags, we wondered if Semantic Versioning still made sense to us. Releasing a feature doesn't require releasing new versions anymore. An automated versioning scheme might suit automated releases better.

Here were some of our goals.

-   Keep the same format like `[X].[Y].[Z]`
-   Keep the version as simple as possible
-   Increment the version for each release
-   Automate versioning

All things considered, we took a simple approach based on time.

-   X: The last two figures of the current year
-   Y: The current month
-   Z: An incremental number that is bumped on each release and reset at the start of a month

For example, if it's February 2022 and the last tag is `release/22.2.6`, the next version would be `22.2.7`. If it's February 2022 and the last tag is `release/21.12.4`, the next version would be `22.2.0`.

We also create a tag like `release/X.Y.Z` for each release. That's it.

Here is the simplified logic in `Fastfile`.

```ruby
lane :bump_version do
  sh('git fetch --tags --quiet')
  latest_version = Gem::Version.new(last_git_tag(pattern: "release/*").split(/\//).last)
  latest_major = latest_version.segments[0]
  latest_minor = latest_version.segments[1]
  latest_patch = latest_version.segments[2]

  if !latest_major || !latest_minor || !latest_patch
    UI.user_error!("Unexpected version number: #{latest_version}")
  end

  year = Date.today.strftime('%y').to_i
  month = Date.today.strftime('%-m').to_i

  major = year
  minor = month
  patch = latest_major != year || latest_minor != month ? 0 : latest_patch.next

  increment_version_number(
    version_number: "#{major}.#{minor}.#{patch}",
    xcodeproj: "Todoist.xcodeproj"
  )
end
```

When it comes to a build number, it's generated based on the current time.

```ruby
lane :bump_build do
  increment_build_number(
    build_number: Time.now.to_i
  )
end
```

One thing to note is that we don't check either the version or the build number into source control. Both are generated on the fly during deployment. If CI is unavailable, we can do that locally as well. Not having to commit those changes to our repositories simplifies the deployment process.

## Release notes

It is important to get early feedback. Internally, we share concise and easy to understand release notes so that everyone at Doist knows what's new. To generate them, we don't rely on pull request titles or commit messages, since this can be time-consuming, or not very meaningful for our team. Instead, we add a changelog to each pull request focusing on behavior rather than technical details, and collect them all using GitHub Actions.

The following is a part of our pull request template.

```md
## Changelog





```

We fill in the information with a reference (`user_facing`, `doist`, and `dev`), depending on the target group.

How do we let our team know about release notes? We send them to Twist. First, we push a git tag after uploading a release build. It triggers a workflow that generates release notes by comparing it with the previous release using [actions/github-script](https://github.com/actions/github-script). Then, we send it to Twist and publish it as a release on GitHub.

<figure>
    <img
        src="/posts/continuous-deployment-for-ios/release-notes-on-twist.png"
        alt="Release notes on Twist"
    />
    <figcaption style={{ textAlign: 'center' }}>
        Todoist iOS Internal Releases thread in Doist Releases channel
    </figcaption>
</figure>

<figure>
    <img
        src="/posts/continuous-deployment-for-ios/release-notes-on-github.png"
        alt="Release notes on Github"
    />
    <figcaption style={{ textAlign: 'center' }}>Releases in Todoist iOS repository</figcaption>
</figure>

For release notes in the App Store, we rely on our changelog team to make them even more user-friendly and give our customers the right level of detail. In the future, we hope to find a more streamlined and automated way.

## Syncing translations

We use [Transifex](https://www.transifex.com) as our localization platform at Doist. Syncing translations is automatically done from the main branch. We have a workflow that runs **every day**. It pushes source files and pulls all translations.

```ruby
desc "Push source files (en) to transifex."
lane :translations_push do
  sh "pip3 install \"transifex-client>=#{min_transifex_client_version}\" && tx push -s"
end

desc "Pull all translations including unreviewed translations from transifex."
lane :translations_pull do
  sh "pip3 install \"transifex-client>=#{min_transifex_client_version}\" && tx pull -f"
end

desc "Commit updated and new translations to sync."
lane :translations_sync do
  translations_push
  translations_pull

  # Commit new translations
  git_add
  git_commit(path: "*.strings", message: "[fastlane] Pull new strings", allow_nothing_to_commit: true)
  git_commit(path: "*.stringsdict", message: "[fastlane] Pull new stringsdict", allow_nothing_to_commit: true)

  git_commit(path: ".tx/config", message: "[fastlane] Pull new transifex config", allow_nothing_to_commit: true)
end
```

To import the lanes, we use the [import_from_git](https://docs.fastlane.tools/actions/import_from_git/) action in `Fastfile`.

```ruby
import_from_git(
  url: "git@github.com:Doist/doist-apple-configs.git",
  dependencies: ["Fastlane/Transifex/Fastfile"],
  ...
)
```

Lastly, a bot creates a new pull request with this change when needed using [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request).

<figure>
    <img
        src="/posts/continuous-deployment-for-ios/sync-translations-pull-request.png"
        alt="Pull request to sync translations"
    />
</figure>

After checks pass and a reviewer approves the pull request, it gets merged. The only thing we have to be careful about is not exposing untranslated content to customers by leveraging feature flags as we already mentioned.

## Managing metadata and screenshots

If an iOS app supports multiple languages in the App Store, metadata and screenshots can be hard to manage. One way of managing those resources is to use Fastlane [deliver](https://docs.fastlane.tools/actions/deliver/).

As it allows specifying paths for metadata and screenshots, there are a few options. The simplest is to directly have them in an app's repository.

Using a separate repository and adding them as a git submodule is another option. We did this for a long time, but it was painful to keep the submodule up-to-date, even with tools like Dependabot.

With the introduction of CD, updating them became even more challenging. We had to keep track of them with a version tag and to update them before automated deployment. Instead, we wanted to separate metadata and screenshots completely from our deployment process. Having a smooth deployment process was critical.

As a result, we have a separate repository to upload metadata and screenshots to App Store Connect via deliver's actions.

```ruby
desc "Upload metadata (without screenshots) to the App Store"
lane :upload_metadata do
  deliver(skip_metadata: false)
end

desc "Upload screenshots (without metadata) to the App Store"
lane :upload_screenshots do
  deliver(
    skip_screenshots: false,
    overwrite_screenshots: true,
  )
end
```

With this, we don't update any resources directly in App Store Connect. To edit them, we commit changes to git and upload them via the actions.

## Managing provisioning profiles and certificates

If you work with multiple developers, managing provisioning profiles and certificates can be painful. We use Fastlane [match](https://docs.fastlane.tools/actions/match/) to simplify code signing. Following the guide from [codesigning.guide](https://codesigning.guide), we have a private repository to sync our provisioning profiles and certificates.

To make use of Xcode's automatic code signing, we rely on it in a local environment, and disable it when uploading a release build.

```ruby
lane :deploy_production do |options|
  sync_code_signing
  disable_automatic_code_signing(path: "Todoist.xcodeproj")
  build_app
  enable_automatic_code_signing(path: "Todoist.xcodeproj")
  upload_to_app_store
end
```

We also need to specify the generated match env variable in Provisioning Profiles.

<figure>
    <img
        src="/posts/continuous-deployment-for-ios/managing-provisioning-profiles.png"
        alt="Managing provisioning profiles"
    />
</figure>

The only thing we do manually is to handle expired or revoked certificates. Currently, it's not possible to automate it easily, so we have a detailed guideline in our internal documentation. It's not a frequent issue, but we're looking for better ways to manage it.

## Rotating release manager

At Doist, we have a [hero role on each team](https://blog.doist.com/heroes-housekeeping-days/). On the iOS team, the hero is responsible for handling issues and doing the manual work for releasing the apps.

With this approach, we're getting feedback from different points of view. For example, a less experienced release manager might question some of the established processes and habits. The key to having better CD is to continuously improve based on feedback from the team itself.

## Conclusion

In this post, I described our approaches to make our deployment faster and easier as a result of adopting Continuous Deployment. This improves our approach to product development, as it incentivizes us to work in a way that makes us proud and passionate about what we do. Each team is different, but I hope you get some ideas of how to improve your own deployment workflow from this post.

Hiroki Nagasawa

Making our deployment better with continuous improvements

Continuous Deployment for iOS


Historically, we haven't shared much logic across platforms at Doist. Features have specs, but each team often implements them separately. Since we build two products, each released in multiple native platforms, we really benefit from sharing code.

While exploring options for multiplatform implementations, [Kotlin](https://kotlinlang.org/docs/multiplatform.html) stood out. It's a language we already use to build our Android apps, and [Kotlin Multiplatform Mobile](https://kotlinlang.org/lp/mobile/) did well in a recent experiment we conducted for iOS. The next logical step was to try it out on the Web, but it was less flawless. This post covers our experience and takeaways integrating a Kotlin/JS library into an existing Web application.

## The library: Filterist

Filterist powers the natural language recognition behind [Todoist's filters](https://todoist.com/help/articles/introduction-to-filters). It receives a query string and returns the tasks that match the query. For instance, for a query like this:

```
(today | tomorrow) & @work
```

It returns all the tasks scheduled for today or tomorrow tagged with the `work` label. It’s implemented in Kotlin on Android, Swift on iOS, and JavaScript on the Web.

We decided this was a good stress test for Kotlin Multiplatform, since the filtering logic is rather complex, with a high amount of supported keywords and operators. Additionally, its domain is mostly business logic, and the output is the same independently of the platform where it runs.

<h2 id="kick-off-goals">Kick-off & goals</h2>

The Kotlin Multiplatform version of Filterist had been architected with the mobile platforms in mind until this point. We jumped into this initiative without knowledge of the programming language and ecosystem. It was time to roll up the sleeves and get acquainted with it. Our goals were:

1.  Minimal changes in native platforms
2.  Effortless to maintain and scale
3.  Small library footprint

Knowing that the Kotlin library was generating versions for Android and iOS that were (almost) fully integrated with the product’s codebases, we presumed that building the JavaScript bundle and integrating it was good enough.

We wrapped the bundle in an npm package, set it as a dependency, and created a test case to assert some natural language use cases. Unsurprisingly, the bundle wasn’t usable right out of the box.

## No public API and types

We installed the npm package, but what was the API? What should the JavaScript code call?

Our Web products use TypeScript to enforce typing and establish an API for the codebase, and Kotlin generates `.d.ts` files containing type definitions along with the JavaScript bundle.

And yet, the file was empty.

We missed that we had to annotate Filterist's public API functions with [@JsExport](https://kotlinlang.org/docs/js-to-kotlin-interop.html#jsexport-annotation). Bingo! After adding those annotations, the main library’s entry point function was listed in the `.d.ts` file and exported by the bundle. The test suite could finally import the appropriate library parts.

## Conflicting platform architectures

We defined a test case that used Filterists entry point function, `filter()`. It had the following type definition:

```typescript
export function filter<T>(
    query: string,
    projects: FilterableProjects<FilterableProject>,
    sections: FilterableSections<FilterableSection>,
    items: FilterableItems<T>,
    user: FilterableUser,
): kotlin.collections.List<Filterist.Result<T>>
```

While some of the arguments had types as `FilterableProject` or `FilterableItems`, none of these were exported or listed in the type definitions file.

The Kotlin function had this structure:

```kotlin
fun <T : FilterableItem> filter(
    query: String,
    projects: FilterableProjects<FilterableProject>,
    sections: FilterableSections<FilterableSection>,
    items: FilterableItems<T>,
    user: FilterableUser,
): List<Result<T>>
```

Those custom types were interfaces. Filterist expected arguments like `projects`, `sections`, or `items` implementing specific interfaces. For example, `FilterableProject` defined a few properties:

```kotlin
interface FilterableProject : FilterableModel {
    val filterableName: String
    val filterableParentId: FilterableId?
    val filterableChildOrder: Int
    val filterableIsShared: Boolean
}
```

On iOS, the `Project` class, which models the state of a project, then extended from `FilterableProject`, deriving the added properties values from properties of the base class.

```swift
extension Project: FilterableProject {
    public var filterableId: FilterableId {
        return globalID
    }
    public var filterableParentId: FilterableId? {
        return parent?.globalID
    }
    ...
}
```

It’s almost plug-n-play. Classes extend these interfaces and provide the necessary properties within state class objects. Since both platforms share a similar state architecture principle, where classes model main business logic entities, it's a sensible choice.

But on the JavaScript side, we use [redux](https://redux.js.org/). Classes do not model the state. Immutable plain objects do. In the end, there was a clash between object-oriented and functional architectures.

Adapting on the JavaScript side would require creating a class layer that extends Kotlin’s interfaces and is passed to the library, maintaining an OOP state layer parallel to the functional redux paradigm. We ruled out this option quickly as we moved away from OOP in our JavaScript codebases years ago. Also, converting each `redux` state property to plain objects that implement Kotlin’s interfaces would be a memory and performance nightmare since there might be hundreds, perhaps thousands, of properties to convert. Lastly, it failed our [first goal](#kick-off-goals).

Instead, we made Filterist more flexible and agnostic.

Some drafts later, and we had the concept of data providers. Rather than passing class modeled state to the library, its functions receive data providers containing getter functions. We defined data providers through interfaces:

```kotlin
interface ProjectAccessorProvider {
    fun getName(projectId: String): String
    fun getParentId(projectId: String): FilterableId?
    fun getIsShared(projectId: String): Boolean
}
```

While each platform imports these interface definitions and creates class or plain object instances that implement those interfaces.

```typescript
const state = getReduxState()
const projectAccessorProvider: ProjectAccessorProvider = {
    getName(id: string) {
        return state.projects[id].name
    },
    getParentId(id: string) {
        return state.projects[id].parent_id
    },
    getIsShared(id: string) {
        return state.projects[id].is_shared
    },
}
```

Filterist no longer cares about state modeling and instead delegates that work to data provider functions. Platforms no longer have to ensure that business logic entities map to Filterist’s interfaces. They only provide getters, which is easy to do as each platform knows how its state is modeled and should be accessed.

## Non-mapped Kotlin types

While the architecture puzzle got solved, it came to light that there was another bump in the road, this time with something that we had less control over.

Kotlin [maps only some of its types](https://kotlinlang.org/docs/js-to-kotlin-interop.html#kotlin-types-in-javascript) to their JavaScript counterparts. One can call Kotlin’s JavaScript exported library functions and pass them JavaScript native types, as `Number`, `String`, or `Array`. Unfortunately, we implemented Filterist using a few non-mapped Kotlin types, such as `List` or, more importantly, `interface`.

Because `interface` doesn’t map to any JavaScript/TypeScript type, the compiler throws an error when an interface is annotated with `@JsExport`. Since Kotlin doesn’t export them, they don’t have TypeScript mappings. Manually adding type information is possible, but it would easily break with new changes to Filterist. Without type information altogether, the only way to venture through the library API would be to read the docs, inspect or debug the generated JavaScript bundle, or dive into the library’s Kotlin implementation.

All would be fine, but none scales well.

### Meanwhile, elsewhere at Doist

In parallel, another Doist team started experimenting with a different library to manage user permissions. They found that it’s possible to [declare external interfaces](https://kotlinlang.org/docs/js-interop.html#external-interfaces) at the package level in the JS-specific source set, making them exportable when annotated with `@JsExport`. There was a catch: external interfaces cannot extend common package interfaces, and other source sets cannot access them.

Let’s see them in action.

In the project’s JavaScript source set, there’s an external interface declared for the relevant state and a function. The latter has the sole purpose of receiving the state (compliant with the external interface) from JavaScript and calling the common package relevant function, passing an object that implements the external’s interface common package counterpart:

```kotlin
external interface ChannelData {
    val userCount: Int
    val isArchived: Boolean
}

fun channelPermissionOf(data: ChannelData): (Int) -> Boolean {
    val permissions = permissionsOf(
        object : CommonChannelData {
            override val userCount: Int get() = data.userCount
            override val isArchived: Boolean get() = data.isArchived
        }
    )
}
```

That logic generates the expected TypeScript definitions for an interface and a function:

```typescript
export interface ChannelData {
    readonly userCount: number
    readonly isArchived: boolean
}
export function channelPermissionOf(data: ChannelData): (p: number) => boolean
```

We learned that it isn’t impossible to overcome the non-exportable interfaces problem. After all, the permissions manager library makes it work without them just fine. However, we decided that this solution doesn’t suit Filterist.

Adopting external interfaces requires creating a one-to-one mapping between external and common package interfaces, and there are lots of them! But doing so means keeping a carbon copy of common interfaces for JavaScript specifically, which fails to meet our [second goal](#kick-off-goals).

### List, interface… and Long

Kotlin's number mapping is another troublesome area for which there might not be JavaScript equivalents. JavaScript's `Number` is a 64-bit double-precision floating-point number, just like Kotlin's `Double`. Its precision for whole numbers ranges from -2<sup>53</sup>+1 to 2<sup>53</sup>-1. However, it maps to Kotlin's `Int`, a 32-bit integer.

In addition, there is no mapping for `Long`, Kotlin's 64-bit integer, so the JavaScript bundle doesn’t make it publicly available. However, the bundle functions expect `Long` arguments because Filterist’s Kotlin code uses it. We experimented with three ideas to overcome this:

#### Use Kotlin.js standard library

The Kotlin.js standard library is publicly available as an [npm package](https://www.npmjs.com/package/kotlin). We installed it, instantiated its `Long` object, and passed it to Filterist’s `filter` function in our JavaScript test case. It didn’t work.

In the generated bundle, Filterist called `Long`’s `plus_wiekkq_k$()` method to do an arithmetic operation. Sketchy naming, uh?

```javascript
Long.prototype.plus_wiekkq_k$ = function (other) {
    return add(this, other)
}
```

That's because Kotlin [mangles](https://discuss.kotlinlang.org/t/implementing-an-interface-causes-name-mangling-on-js-target/196) names to support function overloading, a concept that doesn’t exist in JavaScript. We had to annotate the Filterist functions exported to JavaScript with [@JsName](https://kotlinlang.org/docs/js-to-kotlin-interop.html#jsname-annotation) to prevent this from happening. However, we don’t control the internal `Long` class. Its methods’ names are mangled, and we can’t annotate it with `@JsName`. Alternatively, instantiating `Long` objects through the standard library npm package kept the object properties and methods names intact. We couldn’t use the standard library to create unmapped object types.

#### Change the argument type to Int

In an attempt to build a proof of concept, we converted all exported functions `Long` arguments to `Int` and added `.toLong()` calls where needed in Filterist’s Kotlin project.

Filterist’s logic was now dependent on JavaScript limitations. Besides, other platforms' integers are 32-bit and not 64-bit values like JavaScript’s `Number`, which would lead to bugs over time. It was acceptable for exploration purposes but failed our [second goal](#kick-off-goals).

#### Create a custom type

We came up with a common custom type, `InteroperableLong`, allowing for type mapping and keeping the `Long` ordeal away from the platform codebases. `InteroperableLong` is defined by an expected declaration in Filterist:

```kotlin
expect class InteroperableLong {
    fun toLong(): Long
}
```

With the actual implementation in each platform. For instance, in JavaScript:

```kotlin
actual class InteroperableLong(private val number: Number) {
    actual fun toLong() = number.toLong()
}
```

Or iOS:

```kotlin
import platform.darwin.SInt64

actual class InteroperableLong(private val int64: SInt64) {
    actual fun toLong() = int64
}
```

With all `Long` occurrences swapped to `InteroperableLong`, the solution was ready for battle testing. Using `toLong()` wherever `InteroperableLong` is used is an extra step compared to having `Long` natively mapped, but without a common type between all platforms, it scales well and is at least a long-term solution.

## Kotlin’s date/time library

[kotlinx-datetime](https://github.com/Kotlin/kotlinx-datetime) is Kotlin's official library to manage dates and times. It encapsulates platform-specific date/time logic in a common API, relying on native libraries under the hood. In JavaScript's case, on [js-joda](https://js-joda.github.io/js-joda/).

Our JavaScript codebases depend on [moment](https://momentjs.com/) for date/time handling, so we explored if `js-joda` could be an alternative within our requirements. Timezone handling is where `moment` shines for us, through [moment-timezone](https://momentjs.com/timezone/). Otherwise, we resort to native date helpers more often than not. `js-joda` includes `moment-timezone`’s dataset, providing the same features that we need from `moment`. Swapping libraries wouldn’t be a daunting task.

`moment` and `moment-timezone` out, `js-joda` in, problem solved. Almost.

There was one last thing. The library wraps `js-joda` in custom classes. This means that JavaScript date/time objects created through `js-joda` npm package instances have a different type than the date/time objects that the generated JavaScript bundle expects. To work around this, we added adapter functions that create instances of the custom classes. Here’s an example that returns `js-joda`’s `Instant`:

```kotlin
import kotlinx.datetime.Instant
import kotlin.js.JsExport

@JsExport
fun createInstant(date: String): Instant {
    return Instant.parse(date)
}
```

Admittedly, it isn’t very refined, seems redundant, and we have to call it for every single `js-joda` object that’s passed to the bundle. We would also prefer to control which date/time dependency Kotlin uses, to improve the odds it fits other projects uneventfully. But for validation purposes, it works.

## Bundle size

On the Web, the size of application assets matters. Fast page loads and efficient resource usage depend on it.

The generated JavaScript bundle includes the [Kotlin standard library](https://kotlinlang.org/api/latest/jvm/stdlib/). We can rely on Kotlin’s [dead code elimination feature](https://kotlinlang.org/docs/javascript-dce.html) (DCE), which performs tree shaking to build much slimmer production bundles. However, we still noticed a several-fold increase in size when comparing both implementations. The native implementation weights at around `10kb`, while the bundle generated by Kotlin, with DCE enabled, measured `41.05kb` (both gzipped).

DCE mitigates the bundle size problem, but it isn’t good enough to be negligible. No benefits outweigh an asset bundle that becomes too large. We expect tradeoffs, but not any tradeoffs.

## Conclusions

Having one implementation shared across platforms is very tempting. It ensures that all platforms have the same requirements implemented and tested, and optimizes human effort to build and maintain. For core business logic like Filterist, Kotlin Multiplatform is very promising.

Our attempt at integrating with JavaScript led to various challenges, the most critical being the lack of support for some core Kotlin types in JavaScript, namely `interface`, `List`, and `Long`. Sadly, we feel that Kotlin/JS is not quite production-ready for Filterist. The workarounds required to make basic interoperability work are at odds with the goals we set at the start of the exploration. However, the permission management library, a smaller greenfield project, tailored itself to Kotlin’s JavaScript limitations and was more successful, reaffirming our conviction on multiplatform libraries.

The rest of the topics were either solved or had manageable downsides.

A failed integration doesn’t mean that all this work was in vain for us, though. At first, we built Filterist’s Kotlin library against our Android and iOS architectures and tailored its design to them. This exercise helped refactor Filterist to be more modular and state agnostic. It’s an important lesson to remember when building shared libraries. More so, this exploration allowed us to have a clear perspective of the state of Kotlin/JS. It doesn’t suit our needs at this point, but it shows promising indicators that it can become a relevant actor in our codebases.

## What’s next

Kotlin is a pillar of our Android codebases and integrates well with iOS. Our Kotlin Multiplatform projects will keep evolving, even if the Web continues using native JavaScript implementations.

We weren’t the only ones that had difficulties with the JavaScript integration. Kotlin’s ecosystem has a very active community, and other developers have reported JavaScript’s mapping shortcomings on multiple occasions ([1], [2], [3], [4]).

The good news is that the Kotlin team marked the interface issue as fixed in the upcoming `1.6.20` release ([1]), which we considered to be the most pressing blocker. It’s still unknown if the remaining non-mapped types will ever be officially available on JavaScript, but the folks over [Deezer](https://deezer.io/) have taken matters into their hands with [KustomExport](https://github.com/deezer/KustomExport). We looked into it, but it was still in active development, and we were unable to use it. We’ll keep it on our radar.

As new Kotlin releases come out, we'll re-evaluate. Other in-house libraries may start to use Kotlin Multiplatform on JavaScript, as the permission management library did. For now, we expect its usage to be sparse until it checks all the boxes for larger projects.

[1]: https://youtrack.jetbrains.com/issue/KT-45434 'KJS: "WRONG_EXPORTED_DECLARATION" when using `JsExport` on interfaces'
[2]: https://youtrack.jetbrains.com/issue/KT-34995 'JS: List, Map, and Set types are hard to use from JS because of mangled member names'
[3]: https://youtrack.jetbrains.com/issue/KT-44494 'KJS / IR: Allow enum classes to be exported'
[4]: https://youtrack.jetbrains.com/issue/KT-45427 'KJS: Interoperability for `Map` collection type'


Pedro Alves

Breaking native boundaries by using shared code

Kotlin Multiplatform on the Web


While in college, I was the typical student who ticked all the boxes in the evaluation form, knowing I would never have to revisit the code I was crafting.

After graduating, I was fortunate to work on a couple of greenfield projects, where I was part of all architectural and product decisions. Besides building products that ticked all functional boxes, I wanted them to be something that my future self would be happy to deal with.

Then, I moved to a company that had an established product with years of development. I had to do a myriad of tasks, from fixing support-reported bugs to convert legacy components from a soon-to-be deprecated technology to the new one everyone’s excited about.

All of these realities really highlighted the importance of having a codebase that not only works, but is also easy to maintain and onboard new people into.
**
With more and more software engineering jobs becoming remote and async, you won’t always have a chance to clarify questions synchronously with your colleagues. You want to set up your code for people to be independent and be able to “fly solo”.

Here’s a set of simple tips that really make a difference on building a codebase that’s easy for everyone to get onboard into, understand and maintain. They’re as easy to apply as they are easy to forget, so always keep them in mind! Let’s dive in. 🚀

# Convention everything

**Have your code as consistent as possible, from architecture, to naming, to code styles.** Whether the approach is perfect or just good enough, what’s most important is that your codebase is consistent.

It’s critical to be able to *learn once and apply everywhere* because:

- Everyone’s background and habits are different, so if you have established patterns, there will be less friction and less subjectivity when discussing solutions.
- It allows focusing on the *content*, not the *form.*
- When someone new joins your project, they won’t have to learn different patterns to understand different parts of the codebase, they just need to learn once, and they’re ready to go.
## Examples

**Layout names**

Follow a structure for your layout names that showcases both technical and functional details, e.g.:

<figure>
    <img src="/posts/building-a-newbie-friendly-codebase/name-conventions.png" alt="Layout names comparison" />
    <figcaption style={{ textAlign: 'center' }}>
        Comparison of uncontextualized with standardized layout files names.
    </figcaption>
</figure>


**Patterns**

Let’s define an example of handling a button click. You can do it in several ways, like:
```kotlin
buttonToChangeColor.setOnClickListener { button ->
    if (button.color == Color.RED) {
        button.color = Color.BLUE
    } else {
        button.color = Color.RED
    }
}
```

or

```kotlin
buttonToChangeColor.setOnClickListener { button -> updateColor(button) }
```

Which one is better? It depends. But if you’re consistent, it’ll be much easier to focus on *what* the click listener is doing rather than *where* or *how* it’s doing it, which is unequivocally more important.

**Code styles**

When code is organized and formatted in the same way, **it’s much easier to concentrate on the** **content** and find what you’re looking for.

When reviewing code or having your code reviewed, there will be less bike shedding and feedback will be more objective and fruitful, e.g., it doesn’t matter if you like *trailing commas* or not, if the convention is to have them, then you just have to follow that and not worry if the reviewer will nitpick on that.

# Information flows in a one-way street

**Have** **a single source of truth and unidirectional data flows.** It’s easier to understand where data is created, propagated to, accessed and which actions can modify it.

Having multiple places creating or updating the same data makes the flow of information and cascading effects hard to understand, creating a highway for bugs that are hard to debug and fix.

# Repel magic *values* 

**Extract all of your “magic values” to constants with clear names.** This will make it easy for *everyone* to understand what those values mean!

An example:
```kotlin
val timeDiff = currentTime - lastLoginTime
if (timeDiff > 259200000) {
    showWelcomeBackMessage();
}
```

What does `259200000` actually mean? Is it days? Minutes? This isn’t clear from the very start.

On the other hand, if you had this:
```kotlin
private const val MIN_TIME_SINCE_LAST_LOGIN_MILLIS = 259200000 // 3 days represented in miliseconds.
    
(...)
    
if (timeDifference > MIN_TIME_SINCE_LAST_LOGIN_MILLIS) {
    showWelcomeBackMessage();
}
```

It’s  much clearer what the value means in that context.

# Use data structures in your favor

Data can be represented in *many* ways. Encapsulating all the attributes under a dedicated class, using just primitives, you name it! That said, some solutions might be more ambiguous or error-prone than others. 

**Always model your data as precisely as you can**.

For instance, if you want to represent the state of a given component, you can do it with a set of strings, e.g.:
```kotlin
const val STATE_LOADING = "state_loading"
const val STATE_LOADED = "state_loaded"
    
// When analyzing the possible values you'd do something like:
val state: String = (...)
when (state) {
    STATE_LOADING -> (...)
    STATE_LOADED -> (...)
    else -> // UNKNOWN STATE.
}
```

But if you do this, the possible values of state might not be clear for everyone, and you don’t guarantee that valid values are enforced, so you need to handle unknown / erroneous cases explicitly, etc. 

A safer alternative to this would be to use, e.g., an enum:
```kotlin
enum class State { LOADING, LOADED }
    
val state: State = (...)
when (state) {
    LOADING -> (...)
    LOADED -> (...)
}
```

With this approach, it’s much clearer what are the possible values for state are, there’s no chance to have values outside the expected domain, and if there’s a typo you’ll have a compilation error. This code is safer to handle and easier to understand. 

For more on this topic, watch this excellent talk by [Christina Lee](https://twitter.com/RunChristinaRun) from [KotlinConf 2018](https://kotlinconf.com/2018/) about [Representing State](https://www.youtube.com/watch?v=-lVVfxsRjcY).

# Test it, test it, test it

Ah, the famous last words: *“We need to ship this ASAP, we’ll add tests later.”*

The truth is that we all acknowledge the importance of tests, but sometimes discard the long-term benefits they offer for the sake of short-term velocity.

Let me tell you a quick (horror) story: not long after having joined [Doist](https://doist.com/), I was working on fixing a bug on [Todoist](https://todoist.com/)’s main task list. The bug was relatively simple, and it had to do with collapsing and expanding *Sections*. I found a way to reproduce the bug and came up with a fix. This fix found its way to production, and I was happy with it. If only I had known that the fix that I was so proud about didn’t go “alone” to production but took a little fellow with it: I ended up breaking collapsing and expanding of tasks. So, when the support reports started coming in, we had to halt the release, do a quick-fix on top of my initial fix and do another release. A major hassle that could’ve been avoided if we had tests that helped catch the bug I unknowingly just introduced.

**When you have tests in place, it’s easier for everyone to catch and understand the collateral effects of what they’re doing and much safer to validate the change's they’re introducing.** 

This is especially significant when people aren’t aware of all the previous considerations a particular piece of code required, like in the example above. If you encourage and foster adding tests for each bug that’s fixed or each feature being added from the very beginning, it’s much easier to keep the ball rolling, to keep your test suite up to date and, most importantly, useful.

# Comments are good, but good code is even better

The feeling towards code comments is similar to tests. Everyone knows they’re useful, but…
There’s always a but. Outdated comments can be very misleading and confuse their readers.

So, even though having comments, in theory, is great, try to reduce them to the bare minimum, so it’s easier to maintain them. **Tests fail loudly but outdated comments fail silently**, so focus on having code that’s easy to read as well as up-to-date and meaningful tests.

You might be wondering what *”code that’s easy to read”* actually means and when you should add comments or not. Of course, it depends on the context and on whoever’s “classifying,” but a few personal rules of thumb are:

- Comments usually explain the *why* of a certain piece of code, not the *how*, as that’s redundant and quickly becomes outdated when you refactor that code.
- If you use established patterns, the reader will most likely have an easier time understanding a piece of code without wishing for comments.
- Use property / method / class / file names that are clear and self-explanatory in their purpose.
- In general, if you’re following this article’s suggestions, you’re on the right track. 😌

# Acronyms are B.A.D.

[**Acronyms Seriously Suck**](https://www.linkedin.com/pulse/acronyms-seriously-suck-lesson-from-elon-musk-nathan-tanner/). They’re only great for people who are comfortable with your glossary. 

Engineers love optimization, so there might be a tendency to write fewer characters and choose “obvious” acronyms in favor of the full version. But the new kid on the block won’t know the glossary and will be left wondering when you say that your PDP has a bug. It’s obvious that you’re talking about your *Product Detail Page*, but someone else may be thinking why on earth their *Personal Development Plan* has a “bug”.

Instead of using `PDP` to refer to your `ProductDetailsPage`, use its full version. Characters are free, and it’ll be clear for everyone what you’re referring to.

# Refactor opportunistically

Keeping a codebase consistent is hard! It also gets harder and harder as time goes by, new patterns get introduced, and people come and go. You have to be intentional about your work. 

**When working on any part of the codebase, always try to leave it better than you found it.** Take that chance to update a parameter name that’s off, to apply that small update you did to the codestyle last quarter, or to update an old component to your new architecture pattern.

Being intentional will get you closer to long-term success than just *“going with the flow.”*

----------

Congratulations on getting to the end of the article! Now you might be wondering: *“Yeah, all of these tips are reasonable, but aren’t they a little obvious? Aren’t they standard?”* Yes, they’re obvious and yes, they *should* be standard. But the fact is that more often than not, they’re not.

So, always keep the following in mind: *“If I was just coming to this codebase, what would facilitate my learning curve?”* and keep on improving it. Incentivize newcomers to give feedback on the codebase, your processes, your techniques, and act on that feedback.

One day, you’ll be the newcomer in someone else’s codebase, and you’ll be grateful they kept this in mind.  💡


Make your codebase easy for everyone to get acquainted with

Building a newbie-friendly codebase


Releasing a new version of an Android app can be a dreaded job for most Android developers. We think:

“Oh no, time for another release… I need to take care of translations, prepare the changelog, wait for the build, test the latest changes, and pray Proguard didn't mess with things it shouldn't. 😩”

<figure>
    <img
        src="https://imgs.xkcd.com/comics/compiling.png"
        alt="'Are you stealing those LCDs?' 'Yeah, but I'm doing it while my Android app builds.'"
    />
    <figcaption style={{ textAlign: 'center' }}>
        <a href="https://xkcd.com/303/">Photo</a> by Randall Munroe licensed under{' '}
        <a href="https://creativecommons.org/licenses/by-nc/2.5/">CC BY-NC 2.5</a>
    </figcaption>
</figure>

But the release should be the best part of our work! It’s the moment we share our hard work with our users. It’s the time we finally give them all those new shiny features and bugs fixes.

“Deployment Frequency” is also a [DORA](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance) metric. The research shows that frequent releases have a positive correlation with team performance. Yes — mobile apps are distributed through third-party stores and developers don't fully control the delivery process. But we can still try to push the borders.

This post explains how we transformed the release process of our [Todoist](https://todoist.com) and [Twist](https://twist.com) Android apps from the old-fashioned multi-manual steps to brisk “continuous deployment”, and how that worked out for us. I hope you can also apply some of those improvements in your deployment process and make it more seamless and fun!

# The old way of doing things

Before, our release process was quite complicated. We released infrequently, with many updates in each release. There were quite a few things to remember (and easy to forget) during the process. Below I'll describe our most significant issues.

## Big branches

The biggest issue we had was long-lasting branches with thousands of code line changes.

Whenever the developer wanted to implement a new feature, they opened a new branch and worked on it in separation till completion. Rebase operations had many conflicts and the review was long with much back-and-forth between author and reviewer.

All this could take months to complete for a moderate-sized feature, and that was against another key DORA metric — "Lead Time for Changes" — the amount of time it takes a commit to get into production.

## Blocking translations

[Todoist](https://todoist.com/) is translated into more than 15 languages and [Twist](https://twist.com/) into 8 languages. So when a release was planned, we first had to merge all related PRs, "freeze" the main branch, push strings for translations and wait a few days till all translators completed their work.

## Building and Testing

Before R8 was introduced, we used Proguard with a heavily customized configuration that squeezed every last drop of performance, making a production build take close to 2 hours.

After that, we manually tested the app, and of course, it wasn't uncommon to find new bugs. After fixing, we had to re-build — another 2 hours of waiting.

Fortunately, R8 decreased the build time to few minutes. This felt amazing! But we still had issues, and finding them just before the release (or even worse - after) was a nasty surprise.

# The new way of doing things

As you can see, that was plenty of manual work. We even created a script that printed a runbook to help the developer follow it. But we didn't stop there.

We've taken a high-level approach, examined best practices and made a plan. Then we've started fixing all issues and root causes one by one.

Here is how.

## Feature flags

The change which had the biggest impact was introducing feature flags. If you've never used a feature flag, it's a way to control the app behavior remotely without deploying a new version. In our apps, it usually looks like this:

_This and the following code snippets have been simplified to keep them concise._

```java
if (FeatureFlag.TASK_DESCRIPTION.isEnabled()) {
    showTaskDescriptionView()
} else {
    hideTaskDescriptionView()
}
```

Now, whenever we start working on a new feature, we set up a new feature flag and merge small pull requests into the main branch frequently. Because we keep the feature hidden from the public, it's not a problem that it's incomplete.

Another significant benefit is the ability to show a feature to a group of selected users and collect early feedback without building a custom APK.

Finally, it also allows us to gradually release a feature and quickly disable it if something goes wrong.

## Automating translations

The second improvement we've made was automating the translation process.

The translation platform we use is [Transifex](https://www.transifex.com/), and for that, we've set up two GitHub Actions.

The first pushes new strings to Transifex whenever there is a change in one of our strings files on the main branch. This action also notifies our translation manager, who further communicates with the translation team and oversees the work. In short, it looks like this:

```yaml
on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  push:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2

      - name: Setup environment
        ...

      - name: Push strings to Transifex
        env:
          TX_TOKEN: ${{ secrets.TRANSIFEX_TOKEN }}
        run: tx push -s

      - name: Ping Localization team
        uses: Doist/twist-post-action@master
        with:
          message: "Todoist strings have been just pushed to Transifex. 🙂"
          install_id: ...
          install_token: ...

      - name: If failed, ping Android team
        if: ${{ failure() }}
        uses: Doist/twist-post-action@master
        with:
          message: "❌ Transifex push action failed"
          install_id: ...
          install_token: ...
```

The second action runs twice a week, and it automatically pulls translations from Transifex.

It also uses our [Android Translations Check Gradle Plugin](https://github.com/Doist/android-translations-check) to verify if translations match the original strings format, so we don't have to check them manually.

The action runs the task and then opens a pull request:

```yaml
on:
  schedule:
    - cron: '0 0 * * 2,4'
  workflow_dispatch:

jobs:
  pull:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2

      - name: Setup environment
        ...

      - name: Pull strings from Transifex
        env:
          TX_TOKEN: ${{ secrets.TRANSIFEX_TOKEN }}
        run: tx pull -f -a --parallel

      - name: Check translations
        run: ./gradlew checkAndroidTranslations

      - name: Open a PR
        uses: peter-evans/create-pull-request@v3
        ...

      - name: If failed, ping Android team
        if: ${{ failure() }}
        ...
```

_**Tip:** Feature flags can be used for parts of the app that still need translations. How? For example, when we implement a new dialog with some text, we hide it behind the feature flag and push it to the main branch. The strings are automatically sent for translations, and when they're translated, the only thing we have to do is to enable the flag (or remove it)._

## Automating the changelog

To automate the changelog, we've written and open-sourced [Changelog Gradle plugin](https://github.com/Doist/changelog-gradle-plugin) to help us manage it. Our changelog consists of a `CHANGELOG.md` file that includes changes introduced with every public release and a changelog directory with unreleased changes. They're structured in files named after their branch and contain a line or two describing the changes.

```
CHANGELOG.md
changelog/
├─ new-feature-1-branch.txt
└─ new-feature-2-branch.txt
```

We also have a GitHub Action that runs the Gradle task to validate the entries and ensure they follow our format. The plugin configuration is:

```kotlin
changelog {
    pendingChangelogDir.set("changelog")
    changelogFile.set("CHANGELOG.md")

    addRule("max length is 72 characters") { it.length <= 72 }
    addRule("cannot end with a dot") { !it.endsWith(".") }
    addRule("should start with a ⭐ or 🐛 or 🕶") {
        it.startsWith("⭐ ") || it.startsWith("🐛 ") || it.startsWith("🕶 ")
    }

    commit {
        val date = LocalDate.now().format(DateTimeFormatter.ISO_DATE)
        prefix = "## $date"
        postfix = ""
        entryPrefix = "- "
        insertAtLine = 2
    }
}
```

During the release, we fire another task from our plugin. It collects all pending entries from the changelog files and appends them to the CHANGELOG.md file together with a link to the PR which introduced the given change.

This changelog gets automatically published to a Twist channel where we share our releases. Later, our copywriters update the public Google Play changelog accordingly.

## Version names

Lastly, we automated version names. We were using [SemVer](https://semver.org/) before, but it's hard to automate and doesn't make sense with continuous deployment.

Why?

When we release tiny changes multiple times per day, every release feels just like a patch. Also, with feature flags, there is no specific version that introduces new functionality so `MAJOR.MINOR.PATCH` scheme cannot be applied anymore.

To solve that, we've switched to simple incremental numbers matching version codes and prepended with `v`, for example, `v8290`. We then tag commits with the release names, and we have [Version name Gradle plugin](https://github.com/Doist/version-name-gradle-plugin) that assigns version name and version code in our `gradle.build.kts` file based on the latest git tag:

```kotlin
plugins {
    id("com.doist.gradle.version-name")
}

android {
    defaultConfig {
        versionCode = project.versionCode()
        versionName = project.versionName()
        ...
    }
    ...
}
```

## Gradle Play Publisher

It's important to mention that we also started to use [Gradle Play Publisher](https://github.com/Triple-T/gradle-play-publisher) by [Alex Saveau](https://github.com/SUPERCILEX) that adds Gradle tasks to publish bundles on Play Store. That's not a trivial project, and the code quality is very high. Thanks Alex!

# Reaping the benefits

The above changes took us some time to implement. Most of them had an immediate impact, but only together they allowed us to start making continuous deployments.

## Automated internal releases on every merge

To kick-off, we have a GitHub Action that makes an internal release whenever the main branch changes. It tags the latest commit with a new version name, builds the bundle file, and publishes the release in the internal track on Play Store. No magic here.

## Weekly public releases with a simple trigger

Our favorite GitHub Action is the one that makes public releases. It's quite complex, but we've split the process into Gradle tasks and wrapped them in a Gradle plugin. We reuse it across our projects, and we can also run the release locally in case GitHub Actions are down.

To be specific, we have two GitHub Actions for public release and one manual step.

### Preparing the release

This action is fired manually. First, it runs our `prepareRelease` Gradle task, which does few things:

1.  it creates a release branch,
2.  merges the pending changelog into the main CHANGELOG.md file,
3.  runs git commit,
4.  finally, it tags the commit with the next version name.

After that, the action builds bundle and apk files and fires the second Gradle task `draftRelease`. This Gradle command does the following:

1.  it pushes the branch and tags,
2.  opens the PR,
3.  drafts the GitHub release.

### Testing

After the first action finishes, we install the apk on our devices and do the last manual smoke test.

Even though we could fully automate the public release process, as we did with internal releases, we decided that we still want to have a final check done by a human before publishing the release. It's a reasonably low-effort task, so it's not a big problem. But we may drop this in the future and have fully automated public releases as well.

### Release

If all works fine, we approve the release PR. That fires the second action, which publishes the bundle files on Play Store and runs one last Gradle task `finalizeRelease` that:

1.  publishes GitHub release draft,
2.  merges release PR,
3.  announces the release in our Twist release channel.

At this point, the release is finished, and the developer can happily go back to coding. 🙌

# What's next

Even though our release process is vastly improved, we're still looking for ways to make it better.

One idea is dropping the manual test and making the release process fully autonomous, as mentioned before.

Another is automating the rollout itself. Right now, our initial rollout is 2%. At this point, we keep an eye on the crash rate and reported bugs. Then, if all looks good, we gradually increase the rollout to 10% and later to 100%. We are planning to automate this soon by periodically firing a GitHub Action that checks the stability of a release, and based on some conditions, increases the rollout.

Also, I haven't mentioned anything regarding automated tests. We don't run our test suite during the release. We run tests on every PR, and assume that tests on the main branch pass but of course, in practice, it's not always the case. There are also issues with flaky tests, but that's a pandora box I don't want to open in this blog post.

So yes, another improvement we could make is a step that runs UI tests against the release build.

# Result

In the end, we're as close to continuous deployments on Android as Google allows. Needless to say, we're delighted with the results.

We've got rid of a scary two-hour manual process and replaced it with an automated setup based on Gradle tasks and GitHub Actions.

As a result, we follow universal best practices, release smaller changes much more frequently, and get value in the hands of our users faster.


Piotr Wilczek

How we setup Android apps continuous deployment

We release our Android apps every day


Do you often find yourself wondering how other developers build specific UI components for their apps, either to replicate them, or to get more knowledge on the technologies used to create them? I did this once, too. And then, I figured out. After this article, so will you.

One of the most useful tools to better understand a UI is the UI debugger in Xcode. It allows us to explore all the UI elements and get all the information related to them. But we can only use it on the apps we develop. We can't use it to debug the UI of third-party apps.

But there's a different way: To debug the UI of third-party apps, we can jailbreak an iOS or iPadOS device and install a package that allows us to debug the UI elements visible on the screen. It works with third-party apps, Apple apps, and even the OS UI.

In this post, we will learn more about Jailbreak and Cydia, how to add a new repository to Cydia, install the package that will allow us to debug the UI of third-party apps, and see some examples. Let's start!

## Jailbreak & Cydia

There's a lot of information online to help you jailbreak your device and install Cydia. I've referred to [this spreadsheet](https://docs.google.com/spreadsheets/u/2/d/11DABHIIqwYQKj1L83AK9ywk_hYMjEkcaxpIg6phbTf0/htmlview#gid=1014970938) maintained by the [r/jailbreak](https://www.reddit.com/r/jailbreak/) community to find the right tool for my device and iOS version.

### Adding a repository

Once Cydia is installed, we need to add a new repository to be able to download and install the package to debug the UI of any app.

1. Open Cydia.
2. Open the "**Sources**" tab.
3. Tap the "**Edit**" button (top right).
4. Tap the "**Add**" button (top left).
5. Enter this URL: `http://nscake.github.io/`.
6. Tap the "**Add Source**" button.
7. Once added, tap the "**Return to Cydia**" button.

### Installing FLEXing

FLEXing is the package that will allow us to debug the UI of any app. The package is based on [FLEX](https://github.com/Flipboard/FLEX), an open-source in-app debugging and exploration tool for iOS and iPadOS.

1. Open Cydia.
2. Open the "**Search**" tab.
3. Search "FLEXing".
4. Open the first search result.
5. Tap the "**Modify**" button (top right).
6. Tap the "**Install**" button.
7. Tap the "**Confirm**" button (top right).
8. Once installed, tap the "**Return to Cydia**" button.

## Start Debugging

Now that you have FLEXing installed, it's time to start debugging.

To activate FLEX, tap and hold the status bar on any screen. It will show the FLEX menu that will allow you to explore the UI.

_**Note:** The [README.md](https://github.com/FLEXTool/FLEX/blob/master/README.md) file of the FLEX repository shows how to use it._

## Examples

There are many interesting applications worth exploring, but as examples, I'll use Reeder, Twitter, and Shortcuts. These appear to be using custom components, and you can validate that assumption using your jailbroken device with the FLEXing package.

### Reeder

Reeder implemented a UI library built on top of `UIKit` to control all the UI elements and interactions shown in the app.

For example, that’s the class hiearchy of their own table view named `Reeder.CoreTableView`:

-   `Reeder.CoreTableView`
-   `Reeder.CoreScrollViewBase`
-   `UIScrollView`

As you can see, their table view is not based on `UITableView`, but on `Reeder.CoreScrollViewBase` which is based on `UIScrollView` from `UIKit`. This allows Reeder to implement their own table view that’s optimized for their own needs.

<video controls>
    <source src="/posts/debugging-the-ui-of-third-party-ios-apps/reeder.mp4" type="video/mp4" />
    <p>
        We can't play the video in your browser.{' '}
        <a href="public/posts/debugging-the-ui-of-third-party-ios-apps/reeder.mp4">
            Watch it here.
        </a>
    </p>
</video>

### Twitter

Twitter uses a custom-built tab bar named `TFNCustomTabBar` instead of `UITabBar`, along with a custom tab bar view controller named `T1TabBarViewController` instead of `UITabBarController`.

This allows Twitter to implement multiple UI elements that aren’t supported by the `UIKit` components, such as:

-   Tab bar item without a title, only icon.
-   Custom tab bar dot.
-   Icon animation when tapped.

<video controls>
    <source src="/posts/debugging-the-ui-of-third-party-ios-apps/twitter.mp4" type="video/mp4" />
    <p>
        We can't play the video in your browser.{' '}
        <a href="public/posts/debugging-the-ui-of-third-party-ios-apps/twitter.mp4">
            Watch it here.
        </a>
    </p>
</video>

### Shortcuts

The Shortcuts composer screen is composed by two main view controllers, `WFEditWorkflowViewController` and `WFActionDrawerViewController` which are managed by `WFComposerPresentingViewController` that _partially_ contains the drawer logic.

<video controls>
    <source src="/posts/debugging-the-ui-of-third-party-ios-apps/shortcuts.mp4" type="video/mp4" />
    <p>
        We can't play the video in your browser.{' '}
        <a href="public/posts/debugging-the-ui-of-third-party-ios-apps/shortcuts.mp4">
            Watch it here.
        </a>
    </p>
</video>

## Conclusion

If you've followed the information shared in this post, you now have a jailbroken device with the FLEXing package installed. Congratulations! Now is the time to dig into the UI of your favorite apps to learn how they were built.


Enric Enrich

Debugging the UI of Third-Party iOS Apps


<i>
    Disclaimer: this is <strong>not</strong> an official project
</i>

I always wanted to build a checklist-type application, which would allow me to quickly write down a list and send a link to someone else. The options boiled down to big applications requiring everyone to set up accounts, or adopting someone else's custom solution. I've even seen people embedding PDFs with checklists, which at least has a great advantage of being print-friendly. As a part of a [personal exploration](https://blog.doist.com/doist-objectives/#different) at work, I built the following application:

<br />
<iframe
    style={{ height: '560px' }}
    src="https://embed-todoist.bloomca.me/embed.html?1:passport:0:0:no:0;2:some%20cash:0:0:no:0;3:credit%20card:0:0:2021-05-14r0t0May%2014:0;4:rainproof%20jacket:0:1:no:0;5:good%20walking%20shoes:0:1:no:1;1:Clothes;p:Travel%20checklist;"
></iframe>
<br />

Go ahead, play with it. It's all running locally on your browser. If you want to keep your changes, you can optionally import your project into your [Todoist](https://todoist.com/) account.

In sum, I wanted to replicate our existing application on a smaller scale. While Todoist provides all the functionality I was interested in, we still require an account to provide a richer experience and additional features. But what I really wanted to build was a much simpler and self-hosted application.

## Bootstrapping the app

Creating an application from scratch was an option, but it would lead to a lot of unnecessary work. Since we already have great UI in Todoist, it was just needed to pick the right components and encapsulate their functionality. Besides, it would be a great test to see how modular our application is (I can gladly confirm, it is highly modular). We're using [React](https://reactjs.org/) to compose our UI from different components, like the task, the scheduler popup, task editor, section header or the task list itself.

Initially, I created a new application and started to copy components 1 by 1 where needed, but it quickly became a rabbit hole to fulfill all relevant imports. Instead, I cloned the application locally and started from scratch by creating a new [Webpack](https://webpack.js.org/) [entry point](https://webpack.js.org/concepts/entry-points/), which became the backbone of the future application. From there, I was able to import only relevant parts and, because the application was cloned, it became much easier to remove or modify unnecessary parts.

The first step was to render a list of tasks, which was relatively easy. We use [Redux](https://redux.js.org/) for our data layer, and it was simple enough to remove properties which we receive from the Redux store, and pass them individually from their parent component: the new application is pretty small, so it made sense to store all the data at the root component. Since I cloned the application, I was able simply to remove all usage of [react-redux connectors](https://react-redux.js.org/api/connect). This worked great!

```js
// I removed existing react-redux bindings
function mapStateToProps(state) {
    return {
        user: getUser(state),
        weekStartDay: getWeekStartDay(state),
    }
}

// and passed properties directly. Some properties,
// like user, were removed completely, because
// the application didn't have users
return <Task weekStartDay={1} />
```

Todoist has a lot of features I had no interest in supporting and ended up removing. Thanks to our code separation and component independence, I was able to remove the ones I didn't need and it solved most of the problems. To use components in different contexts, sometimes it's worth having a wrapper that passes all relevant properties to a simpler component. This way, it's possible to reuse the same low-level component while simply passing different properties.

At this point, I implemented deployment process. For the application to be standalone, all I needed to do was compile all static assets and deploy the resulting folder to a server. I did it early and often, which allowed me to collect early feedback, and it helped a lot with testing and developing the idea further.

## Storing data

Now came one of the most important tasks — how to store data. A primary goal was for the application itself to just be static assets on a server, without any backend work, which both simplifies my life and the application itself: there is no need for security, database, monitoring, and other things to ensure that the service works smoothly. The idea was for the app to let you create a list and then pass a link around to somebody, so that they can immediately see it. The most obvious solution was to leverage a URL shortener service, which would create a short unique URL that I could match to the user's data. However, that would require building a service with database to store shortened URLs and matching full version, and I didn't want to go this route, so I had to look somewhere else.

The only thing that is preserved when you share a URL is the URL itself, which can be [at least up to 2000 characters](https://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers). I really liked the idea of putting information in the URL query parameter. 2000 symbols proved to be quite a lot in my testing. Querying my own tasks, their average length is 20 symbols. Using 20 more symbols for additional metadata, it means this approach would support close to 50 tasks, which is great! So, I decided to serialize data into the URL. To shorten it, I gave up on using JSON to avoid property names, quotes, and other ceremonial symbols: it might look like a small decision, but it saved 15% of space on average. Instead, I just listed all parameters separated by `:`. Later, when starting to add new properties like support for completed tasks, due dates, sections, and nested tasks, it proved to be easily extendable as long as new properties were appended at the end.. The only trick is that I couldn't delete old properties or compatibility would be broken. In my case, I kept all original data and added new, and it worked great. To give you an example, here is a URL with one task:

<div style={{ fontSize: '12px' }}>

```bash
https://embed-todoist.bloomca.me/embed.html?1:passport:0:0:no:0;2:some%20cash:0:0:no:0;3:credit%20card:0:0:2021-05-14r0t0May%2014:0;4:rainproof%20jacket:0:1:no:0;5:good%20walking%20shoes:0:1:no:1;1:Clothes;p:Travel%20checklist;
```

</div>

You can see the data right after the query, with the following structure: `id:content:parent_id:section_id:due:checked;`.

After being able to render a list relying only on the URL, it was time to send it to someone else. They should see the same list! That was a great moment. To illustrate this example, [this link](https://embed-todoist.bloomca.me/?4:outline%20the%20structure:0:1:no:0;5:motivation/why:4:1:no:0;6:conclusions:4:1:no:0;7:technical%20challenges:0:1:no:0;8:examples%20to%20embed:0:1:no:0;9:create%20correct%20SEO%20tags:0:2:no:0;10:schedule%20the%20release:0:2:no:0;1:Draft;2:Publish;p:Blog%20Post;) will open the same application as embedded below:

<br />

<iframe
    style={{ height: '670px' }}
    src="https://embed-todoist.bloomca.me/embed.html?4:outline%20the%20structure:0:1:no:0;5:motivation/why:4:1:no:0;6:conclusions:4:1:no:0;7:technical%20challenges:0:1:no:0;8:examples%20to%20embed:0:1:no:0;9:create%20correct%20SEO%20tags:0:2:no:0;10:schedule%20the%20release:0:2:no:0;1:Draft;2:Publish;p:Blog%20Post;"
></iframe>

## Connecting to Todoist

I was doing pretty good on time, so I decided to expand the scope a bit, and connect it to actual Todoist, not by using the live data, but instead providing the user an opportunity to create a real Todoist project from the current list. To do so, we need to authorize the application to perform actions in Todoist on behalf of the user. Todoist supports [OAuth protocol](https://developer.todoist.com/guides/#oauth), and is a standard across applications requesting permissions. The only problem is that it requires to have a server page, which sends a request to Todoist with client ID and client secret, and they have to be stored on our server so that nobody can access them. This was an extra functionality and not a core feature, and it does not require to maintain a database as well, so I decided to add this additional feature. To make it work, I needed to register my application and save callback URL, and open a new window in the browser with the url "https://todoist.com/oauth/authorize?client_id=${CLIENT_ID}&scope=data:read_write". Todoist will put a special code as a query parameter and redirect to that page, and on our server we'll make a request to "https://todoist.com/oauth/access_token" to exchange the code for actual user token. Here is the backend code to make a request and receive user token:

```js
const postData = querystring.stringify({
    client_id: CLIENT_ID,
    client_secret: CLIENT_SECRET,
    code: query.get('code'),
})

const options = {
    hostname: 'todoist.com',
    port: 443,
    path: '/oauth/access_token',
    method: 'POST',
    headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
        'Content-Length': Buffer.byteLength(postData),
    },
}

let response = ''
const tokenRequest = https.request(options, (tokenResponse) => {
    // save all byte data
    tokenResponse.on('data', (d) => (response += d))

    tokenResponse.on('close', () => {
        // object with `access_token` property
        const jsonResponse = JSON.parse(response)
    })
})
```

As you know by now, the application is embeddable. All I needed was to [detect an iframe](https://stackoverflow.com/questions/326069/how-to-identify-if-a-webpage-is-being-loaded-inside-an-iframe-or-directly-into-t), and add some iframe-specific styles and markup. Here is a function to detect whether we are inside an iframe or not:

```js
function inIframe() {
    try {
        return window.self !== window.top
    } catch (e) {
        return true
    }
}

// After I detected that we are in an iframe
// environment, I added a class to the page:
if (inIframe()) {
    document.body.classList.add('embed-iframe')
}
```

## Embedding Todoist's editor

The last big milestone was to make the task editor work. Potentially, it would be the most challenging part, because the editor supports tons of options, some of which I did not need, like:

-   [assigning a task](https://todoist.com/help/articles/assign-a-task-to-someone-else) to a collaborator
-   moving a task to a different project
-   adding [labels](https://todoist.com/help/articles/introduction-to-labels)
-   adding [reminders](https://todoist.com/help/articles/introduction-to-reminders)

While I did not need these specific features, there were others that were helpful to retain. For example, [due dates](https://todoist.com/help/articles/due-dates-and-times), to make it possible to attach a specific date to tasks. With [natural date parsing](https://todoist.com/help/articles/due-dates-and-times#some-example-date-formats-you-can-use), it became possible to type tasks like _send a letter in 5 days_ and have it automatically pick up the correct due date. To keep scope in check, language support was limited to English.

All these features are connected to the editor via plugins, with properties to enable/disable them depending on the context the editor is rendering in:

```js
<RichTextInput
    ...
    isAssigneeDisabled={isAssigneeParseDisabled}
    isDateistDisabled={isDateParseDisabled}
    isPriorityDisabled={isPriorityParseDisabled}
    isLabelsDisabled={!canUseLabels}
    ...
/>
```

With this, I could easily disable unwanted properties and remove related imports. All features inside the component already checked whether they were enabled or not, which made my job much easier. This design allows fine grained control over the feature set of the editor in specific situations, like this one.

## Conclusion

Overall, it was a great exercise, I got a chance to bend our codebase, which proved to be quite flexible, not a light achievement after 10+ years of development. And I was able to use components I hadn't a chance before. I am really pleased with the result and I am glad I had this opportunity!
As a takeaway, I am very impressed with how modern frameworks make it easy to create components which can be slightly modified and reused in different contexts. And since adding another entry point is very easy, together they facilitate building different applications using a single codebase. In fact, we have used this approach with great success to create new Quick Add page, and a dedicated Electron application.


Technical content authored by engineers, for engineers.