Structured/Unstructured Data

According to whether data has a predefined data model, we can divide it into three categories.

1.1 Structured Data

Definition:
Structured data is highly organized and neatly formatted data that follows a strict predefined schema. It is the cornerstone of traditional relational databases (RDBMS).

• Core features:
  • Two-dimensional table structure: data can fit perfectly into rows and columns.
  • Strong types: each field has a clear data type, such as Integer, Varchar, or Date.
  • Easy to compute: computers can retrieve, aggregate, and analyze it very efficiently through SQL.
• Typical sources:
  • Core enterprise systems: ERP, CRM, financial systems.
  • Transaction records: bank transfers, e-commerce orders.
• Technical extension:

  In big data, structured data usually uses a Schema-on-Write^[1] strategy. The structure must be defined before data is written. This guarantees data consistency, but sacrifices write flexibility.

1.2 Unstructured Data

Definition:
Unstructured data has no predefined model or organization and is usually treated as binary objects (BLOBs). This kind of data accounts for 80% - 90% of global data volume. It is both the main challenge and a major source of value in big data analysis.

• Common forms:
  • Multimedia: images, audio, video.
  • Documents: PDF, Word, design drawings.
  • Machine data: although some logs have a format, they are often treated as unstructured streams before cleaning.
• Processing difficulty:
  It cannot be queried directly with SQL. Usually, OCR (optical character recognition), NLP (natural language processing), or CV (computer vision) is needed to extract features and convert them into structured labels for storage.

1.3 Semi-Structured Data

Definition:
Semi-structured data sits between the two. It has some self-describing structure, but does not conform to the strict table form of relational databases. It organizes data through tags or keys.

• Core technical carriers:
  • XML (Extensible Markup Language): an early Internet data exchange standard.
  • JSON (JavaScript Object Notation): the mainstream format for modern Web APIs and NoSQL databases such as MongoDB.
  • HTML: webpage structural markup.
• Example:
  An HTML document itself is an unstructured text stream, but through tags such as <h1> and <div>, machines can identify headings and paragraphs, giving it semantic structure.

Data Models

Relational data models and non-relational data models.

Relational Data Model

2. Relational Data Model (RDBMS): Classic Order

Relational Database Management Systems are based on mathematical set theory and relational algebra. Since the 1970s, they have been the standard for enterprise applications.

2.1 Core Concepts

In an RDBMS, data is organized into Tables.

Data example: User Table

Non-Relational Data Model (NoSQL)

With the explosion of Web 2.0, traditional RDBMS encountered bottlenecks in high concurrency, massive storage, and scalability. NoSQL (Not Only SQL) emerged in response.

3.1 Four Main Types

NoSQL is not a specific technology, but a technology family. It mainly includes the following four categories:

1. Key-Value Stores

   • Principle: similar to Map/Hash in programming languages, using a key to quickly read and write a value.
   • Examples: Redis, Memcached.
   • Scenarios: caching, session management, real-time leaderboards.

2. Document Stores

   • Principle: store documents in JSON/BSON format with flexible schema (schema-free).
   • Examples: MongoDB, Couchbase.
   • Scenarios: content management systems (CMS), e-commerce product details with highly variable attributes, fast-iterating businesses.

3. Column-family Stores

   • Principle: store data by columns, good at writes and large-scale aggregation queries.
   • Examples: Apache Cassandra, HBase.
   • Scenarios: IoT time-series data, massive logs, user behavior traces.

4. Graph Databases

   • Principle: store nodes and edges, focusing on complex relationship networks.
   • Examples: Neo4j, JanusGraph.
   • Scenarios: social network recommendations, anti-fraud risk-control relationship networks, knowledge graphs.