Datasette Cloud

Introducing Datasette for Newsrooms

We’re excited to introduce Datasette for Newsrooms, a new product suite from Datasette Cloud built specifically for investigative journalists and data teams. It's an all-in-one data store, search engine, and collaboration platform designed to make working with data in a newsroom easier, faster, and more transparent.

Built for how newsrooms work

News organizations today deal with troves of documents, spreadsheets, databases, and leaks. But too often, that data is siloed, hard to search, or dependent on a handful of experts to operate. Datasette for Newsrooms changes that.

With Datasette Cloud you can:

Upload and organize your data: CSVs, SQL dumps, document troves, and more.
Process unstructured data with AI, transforming messy data sources into searchable, filterable tables.
Search and filter quickly with powerful full-text search, filters and facets.
Visualize your data with built-in charts and maps, or export it to your favorite tools.
Run SQL directly in your browser, and share your queries with your team.
Collaborate with your team using comments and mentions.
Publish your findings—entire tables or specific query results—transparently to your audience.

Empower your data nerds

Does your newsroom have one over-worked data expert who everyone else depends on for pulling numbers, cleaning spreadsheets, and answering every question involving data?

Datasette lets those data experts build a self-service portal for the rest of the team.

Reporters can search and filter data, run saved queries and visualize and export the results, independently or in collaboration with each other and the resident data nerds.

Our goal is to making newsroom data workflows more collaborative and less reliant on gatekeepers.

And when it’s time to go to press, you can publish the data behind the story for your readers, helping build trust and transparency in your reporting.

We'll help you get started

Sign up for our trial and we'll give you hands-on help getting started. We'll work with you to import your data and onboard your team. You can pay us in feedback!

We’re offering one year of free access for nonprofit newsrooms, and a two month trial for everyone else.

Reach us at hello@datasette.cloud and we'll get you set up.

datasette-pins — a new Datasette plugin for pinning tables and queries

datasette-pins is a new Datasette plugin that allows you to "pin" specific databases, tables, and queries to your Datasette homepage. It's part of a much larger redesign of the Datasette experience. Try it out!

datasette-pins is a new open-source Datasette plugin that allows you to "pin" items like databases, tables, and queries to your Datasette homepage.

Usually, the Datasette homepage only shows top-level databases and a few tables. But that's not very friendly or easy to use! Typically, you only care about a specific table or query at a time, so we hope these pins can act like shortcuts to get you where you want faster.

datasette-pins can be configured to give specific users "write" permissions, allowing them to pin items. This option can be found in the database, table, and query action menus:

How `datasette-pins` works

This plugin uses the new internal database to store pinned items. This means pins persistent across restarts, as long as you provide a path to an internal database at startup like so:

bash datasette my_data.db --internal internal.db

Pinned items will be stored in datasette_pins_* tables inside internal.db. You will never have to manually edit these tables, the plugin will do all the work for you! This internal database feature is only a part of Datasette 1.0 alpha releases.

We use the new top_homepage() "template slot" to inject new content on top section of the homepage. This is a new plugin hook only available in recent Datasette 1.0 alpha releases.

Let us know what you think!

Try it out and tell us what you think! We eventually want to add more features, like "personal" pins and pinning plugin pages, and customizing how your pins appear. File an issue on GitHub with any bugs or ideas you come up with.

Extracting data from unstructured text and images with Datasette and GPT-4 Turbo

Clean data, with well defined columns and rows, is a beautiful thing - the ideal starting point for any data analysis or visualization project. Sadly, very little of the world's interesting data is published in a structured format that we can start using straight away. datasette-extract is a new tool for Datasette that uses GPT-4 Turbo to create and populate database tables using data extracted from unstructured text and images.

Processing unstructured data is one of the most directly useful applications of Large Language Models - the technology behind tools like ChatGPT and Google Gemini.

OpenAI today made their GPT-4 Turbo with Vision model generally available via their API.

Datasette Extract uses that API to convert text and images into structured table data. Here's a demo video of the new plugin in action:

How to use Datasette Extract

Once the datasette-extract plugin is installed, Datasette will gain a new option in the Database action menu: "Create table with AI extracted data".

This will take you to a page where you can configure the table you would like to create.

Here I'm creating an events table with event_title, event_date, description, venue_name and start_time columns.

All of these are text columns. I've set a hint of YYYY-MM-DD for the event_date column and HH:MM 24hr for the start_time column.

Next, paste data into the box below. This can be completely unstructured - in this case I hit Command-A and then Command-C on the upcoming events calendar for the Bach Dancing & Dynamite Society jazz venue in Half Moon Bay.

Hit that "Extract" button an Datasette will send the text to OpenAI, along with the schema for the table you want to create.

The next screen shows the extracted data as it flows in.

Once the extraction is complete you'll be redirected to the newly created database table:

In addition to pasted text, the plugin can handle images. Here's a screenshot image I took of event information from the Visit Half Moon Bay events calendar:

In the demo I uploaded this image using the "Extract data into this table with AI" table action menu, which meant I didn't have to configure the columns again.

The first time I tried this it didn't quite work: the events in the image did not have a visible year, and GPT-4 Turbo defaulted to assuming they took place in 2019 for some reason.

I fixed this by adding an extra hint to the event_date column telling it to assume 2024. Running the import again successfully extracted the events with the correct dates.

Try this out yourself

The Datasette Extract plugin is available for you to use with the Datasette open source project right now, or get in touch if you'd like to try It out on Datasette Cloud.

datasette-plot - a new Datasette Plugin for building data visualizations

datasette-plot is a new Datasette plugin for making visualizations from tables and SQL queries inside of Datasette. It's a spiritual successor of datasette-vega, with the goal of supporting more data visualization types and better integration with new Datasette features. Try out the demo, or install it on your Datasette instances today!

It seems like everyday there's a new frontend JavaScript framework being invented, but JavaScript data visualization frameworks are a once-in-a-lifetime phenomenon. Sure, there are the famous ones like Vega, Recharts, and Plotly.js, all of which depend on some way on D3, the quintessential low-level JavaScript visualization library. But building re-usable data viz libraries and components is hard, so novel data viz frameworks are rare.

Which is why I was shocked when Observable Plot was released in 2021! Created by the original D3 creator Mike Bostock, maintained by the Observable team. Even if you spend just a few minutes scrolling through the Plot Gallery, you'll soon understand how intricate and beautiful data visualization can be in Plot.

But more importantly - the code used to create these visualization is short! Take for example the Stacked area chart example in Plot. Normally, building stacked area charts in pure D3 would typically be a painful and long experience, filled with bugs are sadness (trust me on this). But in Plot, it basically boils down to a single line:

Plot.areaY(unemployment,  {
    x: "date", 
    y: "unemployed", 
    fill: "industry", 
    title: "industry"
}).plot()

So, as we reviewed what data visualizations options we have in Datasette, we decided to combine Plot's expressive, simple charts with Datasette's collaborative and explorative interface.

The result — datasette-plot!

How datasette-plot works

datasette-plot is a Datasette plugin that can be install with:

datasette install datasette-plot

Once installed, navigate to a table or SQL query in Datasette (or try this demo), and you'll see a new "Show plot" option like so:

Once clicked, you'll be greeted with a default chart with some columns pre-configured.

In this case, the default "visualization mark" is a single Dot plot. The first two numerical columns are used as X and Y, with default black dots.

Let's make it pop with some color! In this Palmer Penguins dataset, we'll change the color of the dots based on which island the penguin lives on.

Your browser does not support the video tag.

And datasette-plot doesn't just allow one visualization mark, we can have multiple! Let's add a linear regression mark for each island group.

Your browser does not support the video tag.

And for good measure, one more linear regression mark for the entire dataset.

Your browser does not support the video tag.

Once you have configured your visualization to your liking, you can share your creation with friends and coworkers with the Link to this plot option right below your chart. That link will encode your visualization options in the URL like so:

http://localhost:8000/test/unemployment?_plot-mark=%7B%22mark%22%3A%22area-y%22%2C%22options%22%3A%7B%22x%22%3A%22date%22%2C%22y%22%3A%22unemployed%22%2C%22fill%22%3A%22industry%22%2C%22tip%22%3Atrue%7D%7D

It's a bit long and unwieldily (may change in the future 👀), but it will link directly back to the same table/SQL query and visualization that you have configured.

Future datasette-plot features

This is initial release of datasette-plot is a small proof-of-concept experiment to see if Plot was a good choice for Datasette. And so far, it is! We plan on making incremental improvements to this plugin to make it an integral part of your workflows. Some of our planned improvements include:

More visualization marks

Observable Plot supports dozens of different visualzation marks, but datasette-plot only uses 4 of them so far! We plan to add more, so let us know which you'd like to see!

Better sharing options

Sharing by URL is great, it would be nice to create re-usable data visualization for use across your tables. Maybe you have a chart that you want to re-use every month, or on different tables with similar schemas. Currently you'd have to rebuild the chart from scratch on every new table, which isn't very fun. Let us know what you think!

Richer data types

datasette-plot defaults to only strings/numbers/nulls when plotting data. The downside: no dates! Observable Plot requires JavaScript Date() objects to properly plot time-series data. Follow #3 for updates on this!

Annotate and explore your data with datasette-comments

datasette-comments is a new Datasette plugin for collaborative discussions on your Datasette. You can leave comments on specific rows of your SQLite tables, @mention your collaborators and coworkers, track your work with hashtags, and more. Check out our demo read-only instance, and try it out today with Datasette Cloud or self-host it yourself!

Data exploration is an inherently collaborative and social process. Rarely do you ever work with data by yourself: eventually you need to bring in a coworker, manager, or rubber duck to make sense of it all.

Datasette, a tool for exploring and publishing data, is already social, to some degree. You can share URLs that link directly to specific tables, rows, and SQL queries of your database. But that requires a "3rd party" platform to share those messages, like a separate Slack, Teams chat, or email thread.

Until now!

datasette-comments for collaborative exploration

datasette-comments is a new Datasette plugin that allows you to communicate with your coworkers or teammates all inside Datasette, without the back-and-forth of Slack or emails. It is free, open-source software that you can run yourself, and an integral feature to Datasette Cloud.

Once setup, you can leave comments on individual rows inside the Datasette UI. Comments live in threads on specific rows, and can be accessed from table or row pages.

Your browser does not support the video tag.

You can "at mention" coworkers, to bring attention to issues, ask for help, or share new insights!

Your browser does not support the video tag.

And even "react" to others messages!

Once your discussion ends, you can "resolve" the entire thread to keep everyone focused.

You can even leave "tags" in comments, to organize your discussions!

Finally, the comments activity page allows you to search through all your discussions, while filtering by comment table location, authors, hashtags, and more.

How datasette-comments works

datasette-comments is a good ol' fashion, open source Datasette plugin. It extensively uses the new internal database feature to store comments. If you make comments on rows inside of your_data.db, the comments themselves are stored inside the separate internal.db database.

The "frontend" requires a lot of interaction, local state, and data updates. So instead of traditional Datasette Jinja templates, we opted for using preact to make the frontend components, and esbuild as the build tool.

Future datasette-comments features

The current datasette-comments release is full of neat features, but is still in beta. We have many feature we want to include in the near future, including a user-facing API, notifications, and more.

An API to create comments

We plan on adding an API to programmatically create comments from outside workflows. For example, you may have a script that automatically create a comment on rows where a column meets a condition (ex dollar_amount > 100.0 or status == 'overdue').

Additionally, you could use the comments API to defer manual triage to a ChatGPT-powered assistant. If it thinks a row has an interesting enough row for human intervention, leave a comment and @ mention you for further review.

Comment notifications

We also aim to improve "notifications" when comments are left, especially when you are mentioned by coworkers. A native inbox inside Datasette would be great, but also integrations with 3rd party services like Slack and emails, so you're always on-top of conversations you're a part of.

Your ideas!

Datasette Comments is open source and built in the public. If you have any feature request, ideas, or suggestion to improve Comments, feel free to start a discussion on Github, or start a conversation in our Discord!

Datasette Cloud

Introducing Datasette for Newsrooms

Built for how newsrooms work

Empower your data nerds

We'll help you get started

datasette-pins — a new Datasette plugin for pinning tables and queries

How datasette-pins works

Let us know what you think!

Extracting data from unstructured text and images with Datasette and GPT-4 Turbo

How to use Datasette Extract

Try this out yourself

datasette-plot - a new Datasette Plugin for building data visualizations

How datasette-plot works

Future datasette-plot features

More visualization marks

Better sharing options

Richer data types

Annotate and explore your data with datasette-comments

datasette-comments for collaborative exploration

How datasette-comments works

Future datasette-comments features

An API to create comments

Comment notifications

Your ideas!

How `datasette-pins` works