Why Does Query Timeout in Automation Studio?

Question

Timeout Error in Automation Studio but Not in Query Studio

I’m encountering an issue where a query runs successfully in Query Studio within a minute but times out in Automation Studio.

Details:

• The target Data Extension (DE) has 4 Primary Keys: SubscriberKey, JobId, BatchId, and Email.
• The query activity’s data action is Update.
• When I run the query in Automation Studio with a cleared Target DE, it works fine. However, when I add approximately 100k records to the Target DE and re-run the query, it times out.

Here’s the query causing the timeout issue:

SELECT          
    s.SubscriberKey,
    s.AccountID,
    s.OYBAccountID,
    s.JobID,
    s.ListID,
    s.BatchID,
    s.SubscriberID as SubscriberID,
    s.EventDate as EmailSentDate,  
    ts.EmailAddr as Email,
    ts.view_email_url as WebURL,  
    SUBSTRING(ts.EmailAddr, CHARINDEX('@', ts.EmailAddr) + 1, LEN(ts.EmailAddr) - CHARINDEX('@', ts.EmailAddr)) as Domain,
    s.TriggererSendDefinitionObjectID,
    s.TriggeredSendCustomerKey,
    o.EventDate as EmailOpenDate,
    o.IsUnique as EmailOpenUnique,
    c.EventDate as EmailClickDate,
    c.IsUnique as EmailClickUnique,
    u.[IsUnique] as Unsubscribed,
    u.[EventDate] as UnsubscribedDate,
    b.[BounceCategory] as BounceType,
    b.[BounceSubcategory] as BounceDetail,
    b.[IsUnique] as Bounced,
    b.[EventDate] as BouncedDate,
    (CASE WHEN b.[IsUnique] IS NOT NULL THEN 'FALSE' ELSE 'TRUE' END) as Delivered,
    cp.[IsUnique] as SpamComplaint,
    cp.[EventDate] as SpamComplaintDate,
    f.[IsUnique] as Forwarded,
    f.[TransactionTime] as ForwardedDate,
    GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time' as CreatedDate                
FROM [_Sent] s
JOIN (
    SELECT 
        JobID,
        BatchID,
        SubID,
        EmailAddr,
        view_email_url
    FROM 
        [TechnicalSendlog]
) ts 
ON s.JobID = ts.JobID 
    AND s.BatchID = ts.BatchID 
    AND s.SubscriberID = ts.SubID
FULL OUTER JOIN [_Open] o
    ON s.[JobID] = o.[JobID] AND s.[BatchID] = o.[BatchID] AND s.[SubscriberKey] = o.[SubscriberKey] AND o.[IsUnique] = 1 
FULL OUTER JOIN [_Click] c 
    ON s.[JobID] = c.[JobID] AND s.[BatchID] = c.[BatchID] AND s.[SubscriberKey] = c.[SubscriberKey] AND c.[IsUnique] = 1
FULL OUTER JOIN [_Unsubscribe] u
    ON s.[JobID] = u.[JobID] AND s.[BatchID] = u.[BatchID] AND s.[SubscriberKey] = u.[SubscriberKey] AND u.[IsUnique] = 1
FULL OUTER JOIN [_Bounce] b
    ON s.[JobID] = b.[JobID] AND s.[BatchID] = b.[BatchID] AND s.[SubscriberKey] = b.[SubscriberKey] AND b.[IsUnique] = 1
FULL OUTER JOIN [_Complaint] cp
    ON s.[JobID] = cp.[JobID] AND s.[BatchID] = cp.[BatchID] AND s.[SubscriberKey] = cp.[SubscriberKey] AND cp.[IsUnique] = 1
FULL OUTER JOIN [_FTAF] f
    ON s.[JobID] = f.[JobID] AND s.[BatchID] = f.[BatchID] AND s.[SubscriberKey] = f.[SubscriberKey] AND f.[IsUnique] = 1
WHERE SUBSTRING(s.[SubscriberKey],1,3) != '003'
AND (
    (s.EventDate AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
    OR (o.EventDate AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
    OR (c.EventDate AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
    OR (u.EventDate AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
    OR (b.EventDate AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
    OR (cp.EventDate AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
    OR (f.TransactionTime AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time') >= DATEADD(hour, -6, CONVERT(DATETIME, CONVERT(DATE, DATEADD(day, -1, GETDATE() AT TIME ZONE 'Central America Standard Time' AT TIME ZONE 'Pacific Standard Time'))))
)

My Questions:

1. Why does the same query behave differently in Query Studio and Automation Studio?
2. How can I optimize this query or resolve the timeout issue in Automation Studio when working with large datasets?

Keywords: Marketing Cloud, Query Studio, Automation Studio, Query Timeout

Answer

The issue likely arises due to the Update data action in Automation Studio, which can be significantly slower than a simple Overwrite action, especially when the Target DE has a large number of records. Below are potential causes and solutions:

1. Inefficient Query Processing:
   Automation Studio may struggle with complex queries when using Update actions due to additional indexing and record-locking operations. Query Studio processes queries differently and doesn’t execute updates on the DE, which might explain the faster execution.
2. Target DE Structure and Indexing:
   With four primary keys, the Target DE requires more computational effort to match and update records. If the indexing on the Target DE is suboptimal, this could exacerbate the issue.
3. Best Practices for Optimization:
   a. Use Temporary Data Extensions: Instead of updating the Target DE directly, write the query results to a temporary DE using an Overwrite data action. Then, use a simpler query to update the Target DE from the temporary DE.
   Example:

-- Query to temporary DE
SELECT * 
INTO [TemporaryDE]
FROM [YourQueryHere]

This query retrieves all data from a source and writes it into a temporary Data Extension (TemporaryDE). This ensures the query runs without the complexities of updating an existing DE with large datasets.

-- Simpler query for updating Target DE
UPDATE [TargetDE]
SET Column1 = Temp.Column1,
    Column2 = Temp.Column2
FROM [TemporaryDE] Temp
WHERE TargetDE.Key = Temp.Key;

This query uses the data from the TemporaryDE to update the TargetDE. It simplifies the process, as the complex computations are already handled in the previous query.

b. Optimize Date Comparisons: Date comparisons involving multiple time zone conversions may slow down queries. Simplify them where possible by precomputing dates or using consistent formats.

4. Test with a New Target DE:
   Create a new Target DE to rule out indexing or other structural issues with the existing DE.
5. Check Automation Studio Settings:
   Ensure that the query activity timeout settings are not overly restrictive. The default timeout for query activities is 30 minutes, but this can vary depending on your instance configuration.

By implementing these changes, you can mitigate the timeout issue and improve the performance of your query activity.

Summing Up

The query provided is timing out in Automation Studio but runs successfully in Query Studio. Possible causes include inefficient date comparisons, the type of operation (update vs. overwrite), and the size of the target Data Extension (DE). To resolve the issue, consider optimizing date comparisons, using a new DE to eliminate potential indexing issues, or creating a temporary DE for data insertion followed by an update. This approach can help avoid timeouts by simplifying the query operation and improving performance in Automation Studio.

Job-Oriented Salesforce Training with 100% Money Back Guarantee

Our Salesforce Course offers an immersive experience into the Salesforce platform, equipping you with the necessary skills to succeed in the CRM industry. The curriculum includes in-depth coverage of key areas such as Salesforce Admin, Developer, and AI, blending theoretical knowledge with hands-on practice. Through live projects and assignments, you’ll gain practical experience that will enable you to tackle complex business problems with Salesforce solutions. Our experienced instructors ensure you develop the technical expertise and industry insights needed to thrive in the Salesforce ecosystem.

Beyond technical skills, our Salesforce training in Mumbai also offers personalized mentorship, guidance for exam preparation, and interview coaching to give you a competitive edge in the job market. You’ll have access to comprehensive study materials, real-world project exposure, and dedicated support throughout your learning journey. By the end of the course, you’ll not only be ready for certification exams but also armed with practical experience and problem-solving abilities that employers highly value. Kickstart your Salesforce career with us and open doors to a range of career opportunities!

The post Why Does Query Timeout in Automation Studio? appeared first on Salesforce Online Training.