You can bring your data from various sources into an appbase.io
app or cluster using one of the following methods:
This table below can help you choose the right method for your use-case:
Ease of Use |
Supports Automation |
Importing Large Dataset |
Data Sources Supported |
|
---|---|---|---|---|
Dashboard GUI | ✔️ | ✖️️️ | ✔️ | CSV, JSON, Elasticsearch and SQL. |
ABC CLI | ✔️* | ✔️* | ✔️ | Everything above plus MongoDB, Firestore, Kafka, Redis, Neo4J. |
REST API | ✖️️️ | ✔️* | ✔️ | No restrictions. |
Zapier | ✔️ | ✔️ | ✖️ | Typeform, Google Sheets, Twilio, and over 1500+ other apps. |
Importing Through Dashboard
Dashboard offers a GUI for importing data visually. It supports the following formats:
- JSON / JSONL - You can bring in any JSON or newline limited JSON files (or hosted URLs) to be imported using the Importer view of the dashboard.
- CSV - You can also bring in a CSV file (or hosted URL) to be imported.
- Elasticsearch - You can import an index from an Elasticsearch cluster hosted anywhere into your appbase.io cluster's index.
- SQL - You can import a SQL database from Postgres, MySQL or Microsoft SQL Server into your appbase.io cluster's index.
- Shopify - You can import data from a Shopify store into your appbase.io cluster's index. With Shopify, we support options for syncing product, collections, product-collection relationships, meta fields and named tags (a faster alternative to meta fields). Read more about importing from Shopify over here.
You can try the Importer UI live by going to Explore Cluster > Develop > Import
tab within your dashboard.
Below is a GIF showing the import process for a JSON file.
Importing Through ABC CLI
While the dashboard GUI supports the most common formats, we offer the ABC CLI which allows importing data from an extensive list of database sources using the abc import
command.
Key Benefits
- Whether your data resides in Postgres or a JSON file or MongoDB or in all three places, abc can index the data into Elasticsearch. It is the only tool that allows working with all these sources at once or individually:
csv
,json
,postgres
,mysql
,sqlserver
,mongodb
,elasticsearch
andfirestore
. - It can keep the Elasticsearch index synced in realtime with the data source.
Note: This is currently only supported for MongoDB and Postgres.
abc import
is a single line command that allows doing all of the above. It doesn’t require any external dependencies, code configuration is optional (and helps with transforming data before being indexed), and runs as an isolated process with a minimal resource footprint.
Installation
You can install ABC CLI from the latest Github release for your OS.
You can also install this via docker using:
docker pull appbaseio/abc
It is possible to import data from various database sources. See the snippets below that show the import command for each of the supported database.
Postgres
abc import --src_type=postgres --src_uri=<uri> <elasticsearch_uri>
Read more about it over here.
MongoDB
abc import --src_type=mongodb --src_uri=<uri> <elasticsearch_uri>
Read more about it over here.
MySQL
abc import --src_type=mysql --src_uri=<uri> <elasticsearch_uri>
Read more about it over here.
Microsoft SQL Server
abc import --src_type=mssql --src_uri=<uri> <elasticsearch_uri>
Read more about it over here.
Elasticsearch
abc import --src_type=elasticsearch --src_uri=<elasticsearch_src_uri> <elasticsearch_dest_uri>
Read more about it over here.
JSON
abc import --src_type=json --src_uri=<uri> --typename=<target_type_name> <elasticsearch_uri>
Read more about it over here.
CSV
abc import --src_type=csv --src_uri=<uri> --typename=<target_type_name> <elasticsearch_uri>
Read more about it over here.
Firestore
abc import --src_type=firestore --sac_path=<path_to_service_account_credentials> <elasticsearch_uri>
Read more about it over here.
SQLITE
abc import --src_type=sqlite src_uri=./data.db?_busy_timeout=5000
Read more about it over here.
Kafka
abc import --src_type=kafka src_uri=kafka://user:pass@SERVER:PORT/TOPIC1,TOPIC2
Read more about it over here.
Neo4J
abc import --src_type=neo4j src_uri=bolt://localhost:7687
Read more about it over here.
Redis
abc import --src_type=redis --src_uri="redis://localhost:6379/0" https://USERID:PASS@scalr.api.appbase.io/APPNAME
Read more about it over here.
REST API
Appbase.io, by extension of Elasticsearch's REST API offers two ways to index data:
- Document Index / Document Update endpoint - Read more about this in the doc and see an example code snippet here.
- Bulk Index endpoint - Read more in the doc and see an example code snippet here.
You can also see the examples below that show an interactive example of using the bulk API for JavaScript, Go, Python and PHP:
Javascript
Go
Python
PHP
Note
It is recommended to index up to 1 MB of data (~500 documents) at a time (so if you have 50,000 documents, you can split them into chunks of 500 documents and index).
Zapier
Appbase.io's Zapier integration allows you to import data from over 1500+ apps such as Mailchimp, Sendgrid, Twilio, Google Sheets, Typeform into your appbase.io app or cluster.
You can start using the integration from here: https://zapier.com/apps/appbaseio/integrations
If you are new to Zapier, it allows connecting apps you use everyday to automate your work and be more productive. You can read more about Zapier over here.
In this section, we will walk through the process of importing data using this Zapier integration. We will use Google Sheets as the input source.
You can also read about importing data from TypeForm into appbase.io using this Zapier integration over here.
Creating A Zap
You can go to the Zapier editor and create a zap. In the below image, we create a zap for Google Sheets <> appbase.io
.
Adding Your Data In Google Sheets
Add data in your Google Sheets. You can directly copy the data from here.
Configuring Google Sheets
Login with your Google account and once you configure the Google Sheets integration, you should see something similar:
Configuring appbase.io
Next, select appbase.io
from the apps and go to the Create Document
action.
After this step, you need to add your API credentials and authenticate. Connect your appbase.io
account on clicking Add a New Account
under Choose Account
section, where you have to
enter your appbase.io
credentials which you can find here.
You should see something similar:
You can perform below operations through Zapier.
Adding New Record
I am going to call my appbase.io app business
here. The Zapier integration will create the app if it doesn't exist. The next step is to map the columns from Google Sheets to the field names that will be used to store them in appbase.io. It should look something similar to the following:
After clicking on Continue
and after a successful insertion, you will also see an _id
being returned back. This is the unique ID for our record (aka document).
Updating An Existing Document
Another helpful feature is the ability to edit existing documents. Whenever a value from an incoming integration is mapped to an _id
field, the existing document gets updated instead of a new document getting created.