Practice Test #3

Spark SQL array functions are not primarily about working with “a variety of types at once.” They operate on array-typed columns (arrays of a single element type, possibly complex like struct). While arrays can contain complex elements, the benefit is not generic multi-type processing; it’s targeted manipulation of array data (access, explode, transform, filter).

Working within partitions and windows is the domain of window functions (for example: row_number, rank, lag/lead, sum over(partition by ... order by ...)). Array functions do not define partitioning semantics or window frames. You might use arrays in combination with window results, but the partition/window capability itself is not provided by array functions.

Time-related intervals are handled by Spark SQL datetime functions and time-window constructs (date_add, add_months, datediff, timestampadd, window for event-time aggregations). Array functions do not provide interval arithmetic or time bucketing. This option confuses array operations with temporal processing features in Spark SQL.

Spark SQL does not provide an “array of tables” concept for procedural automation via array functions. Automation is typically done with Databricks Workflows/Jobs, notebooks, DLT pipelines, or external orchestration tools. Array functions operate on array columns within a dataset, not on collections of tables as procedural objects.

Incorrect. While Gold tables are often more directly aligned to business use cases, “valuable” is subjective and not a defining property of the Medallion layers. Silver can be extremely valuable as the reusable, conformed foundation for many products. The architecture defines refinement and purpose, not an absolute measure of value.

Incorrect. Gold is generally more refined and more purpose-built than Silver, not less. Silver is the cleaned/conformed layer, whereas Gold is the curated serving layer (often with business logic, dimensional modeling, and aggregates). A “less refined view” would more closely describe Bronze, not Gold.

Incorrect. Gold does not necessarily contain more data than Silver. Because Gold frequently includes aggregations and curated subsets for specific use cases, it often has fewer rows than Silver. Silver commonly retains detailed, conformed records that can feed many Gold tables, so it can be larger in volume.

Incorrect. “Truthful” is not guaranteed by being in Gold. Data quality and correctness depend on validation rules, expectations, and governance applied across the pipeline. Silver is often where many quality controls are enforced; Gold may add business logic and aggregation but is not inherently more truthful than Silver.

Checkpointing stores progress and state information so a streaming query can recover after failures and avoid reprocessing data. It is essential for reliability and exactly-once-style behavior, but it is not the engine Auto Loader uses to process data incrementally. The question asks for the tool used by Auto Loader, and that underlying tool is Spark Structured Streaming.

Data Explorer is a user interface feature in Databricks for browsing catalogs, schemas, tables, and files. It helps users inspect data assets, but it does not ingest files or run incremental processing jobs. Therefore, it has no role as the processing tool behind Auto Loader.

Unity Catalog is Databricks' governance layer for managing data access, lineage, and object organization. It can secure the tables and storage locations involved in an Auto Loader pipeline, but it does not perform ingestion or streaming computation. As a result, it is not the tool Auto Loader uses for incremental processing.

Databricks SQL is designed for SQL analytics, dashboards, and querying data stored in the lakehouse. It is not a streaming ingestion framework and does not power Auto Loader’s file-by-file incremental processing. It may be used after ingestion to analyze the data, but it is not part of the ingestion engine.

trigger("5 seconds") is not the correct PySpark Structured Streaming syntax. The trigger API expects named parameters such as processingTime, once, or continuous rather than a bare positional string. Because of that, this option does not properly express a 5-second micro-batch schedule. On the exam, remember that time-based micro-batching must be written as processingTime="...".

trigger() with no arguments uses Spark's default trigger behavior rather than a fixed 5-second interval. In practice, Spark will run micro-batches as soon as possible based on data availability and system capacity. That means it does not guarantee execution every 5 seconds. Therefore it does not satisfy the explicit scheduling requirement in the prompt.

trigger(once="5 seconds") is invalid because the once trigger is not configured with a time string. The once option is used to run the query a single time and then stop, typically expressed as once=True. It does not create recurring micro-batches every 5 seconds. This makes it incompatible with the requirement for periodic execution.

trigger(continuous="5 seconds") refers to continuous processing mode, which is different from standard micro-batch execution. The question explicitly asks for a micro-batch every 5 seconds, so continuous processing is the wrong execution model. Continuous mode also has different support limitations compared with normal micro-batching. Therefore this option is not the correct way to schedule 5-second micro-batches.

Increasing the cluster size of the SQL endpoint can reduce runtime for CPU/IO-intensive queries by giving each query more resources. However, for many small queries running simultaneously, the dominant issue is often queueing due to limited parallel capacity. A larger single cluster does not always increase the number of concurrent queries proportionally, so latency under high concurrency may remain high.

Auto Stop shuts down the SQL endpoint after inactivity to save cost. It does not improve latency when the endpoint is already always-on and actively serving many users. In fact, if enabled aggressively, it can worsen user experience by introducing cold-start delays when the warehouse has to restart. This option addresses cost optimization, not peak concurrency latency.

Turning on Serverless can improve operational simplicity and can reduce startup time and some management overhead. However, the question’s problem occurs while using an always-on endpoint under high concurrency. The most direct fix is to increase scale-out capacity (max clusters). Serverless alone does not guarantee sufficient concurrency unless scaling settings also allow additional capacity.

Spot Instance Policy settings apply to classic (non-serverless) compute choices and cost/reliability tradeoffs; serverless abstracts infrastructure selection away from the user. Even if interpreted broadly, changing spot policy is not a primary lever for reducing latency from concurrent small queries. The core issue is insufficient scale-out capacity, which is addressed by scaling range rather than spot/reliability tuning.

Cloud-specific integrations are not a core benefit of embracing open source. Open source typically emphasizes portability and broad compatibility across environments. Databricks does provide deep integrations with AWS/Azure/GCP services, but those are platform and cloud partnership features, not a direct consequence of using open source technologies.

Simplified governance is primarily delivered through Databricks platform capabilities such as Unity Catalog (centralized permissions, lineage, auditing) and related governance tooling. Open source alone does not inherently simplify governance; governance depends on consistent identity, access controls, policies, and metadata management across the platform.

Ability to scale storage is mainly a benefit of using cloud object storage (like S3, ADLS, GCS) and decoupling storage from compute. While open formats can help keep data accessible, the elastic scaling of storage is a cloud infrastructure characteristic, not specifically a benefit of open source adoption.

Ability to scale workloads is driven by Databricks’ elastic compute (autoscaling clusters, job clusters, serverless options) and distributed processing (Spark). Spark is open source, but the exam framing asks for a benefit of embracing open source technologies; the more canonical, differentiating benefit is portability and reduced lock-in rather than generic scalability.

The Databricks web application is part of the Databricks control plane, operated by Databricks. It includes the UI and backend services that manage workspaces, jobs, and APIs. Customers access it over the internet or private connectivity, but it is not stored in the customer’s cloud account. This is a common control plane vs data plane distinction tested on the exam.

Cluster management metadata (e.g., cluster definitions, configurations, job/cluster state) is generally stored in the Databricks control plane as workspace/management metadata. While the actual compute resources (VMs/instances) run in the customer’s cloud account (data plane), the metadata about managing those clusters is maintained by Databricks services, not stored as customer-owned cloud storage objects.

Repos are a workspace feature that integrates with external Git providers (GitHub, Azure DevOps, GitLab, Bitbucket). The repo content is synchronized from the Git provider and represented as a workspace artifact. This is not typically described as being stored in the customer’s cloud account; it is managed through the Databricks control plane/workspace layer and the external Git system.

Notebooks are workspace artifacts stored in the Databricks control plane (workspace storage/metadata). Although notebooks can be exported to files or stored in Git via Repos, the default notebook objects you create and edit in the Databricks workspace are not stored in the customer’s cloud account. The exam often contrasts notebooks (control plane) with data (customer cloud storage).

A Spark SQL table is a persisted object registered in a catalog/metastore. Creating a table from two tables (e.g., CTAS) typically writes the result to storage, increasing storage costs. Even if it references external data, it is designed for reuse and persistence across sessions, which is unnecessary given the question’s constraints.

A (persistent) view stores only the SQL definition and does not copy physical data, which matches the storage requirement. However, it is persisted in the metastore and intended to be accessible across sessions (and potentially by other users, subject to permissions). Since the object is not needed beyond the current session, a temporary view is a better fit.

A database in Databricks/Spark SQL is a namespace used to organize tables and views. It does not represent a relational object created by joining two tables, nor does it directly address the requirement to avoid copying data. Creating a database would not provide the requested joined relational object.

A Delta table is a storage-backed table format (Parquet + transaction log) and represents persisted data. Creating a Delta table from a join generally materializes the result and writes files, increasing storage usage. Delta is ideal when you need ACID transactions, time travel, and reliable persistence, but it conflicts with the goal of avoiding physical data copies.

Incorrect. Changing the location of data is not a reason to choose MERGE INTO. Data location changes relate to table definitions (managed vs external), CTAS, deep/shallow clones, or rewriting data with commands like CREATE TABLE ... LOCATION or COPY INTO. MERGE and INSERT are DML operations that write rows; they don’t change a table’s storage location.

Incorrect. Whether a table is external does not determine MERGE vs INSERT. The key requirement is that the target table must be a Delta table to use MERGE INTO. External Delta tables can still be merged into, and external non-Delta tables cannot. INSERT INTO can work for many table types, but external-ness alone is not the deciding factor.

Incorrect. Deleting the source table is unrelated to the choice between MERGE and INSERT. MERGE is chosen based on needing conditional updates/inserts (upserts) into the target. Source lifecycle management (dropping staging tables, cleaning files) is an operational concern and can apply regardless of whether you used INSERT or MERGE.

Incorrect. The source does not need to be a Delta table for MERGE INTO; it can be a view, a subquery, or data read from Parquet/CSV and transformed into a DataFrame/view. The critical requirement is that the target table is Delta so it can provide ACID transactions and row-level operations needed for MERGE.