Dedupe

Dedupe enables you to identify and merge large quantities of duplicate records in both standard and custom CRM objects. 

• Choose from a variety of pre-built deduplication scenarios or create your own
• Use a broad array of comparison types to more easily identify duplicate records  
• Deduplicate previously identified duplicates from an external file
• Determine master records based on your company's logic or use pre-defined master rules 
• Control field values on winning records using robust field rule functionality 

Starting a Dedupe Scenario

To begin a new scenario, select Dedupe from the module list.

To start from an existing scenario, select a Dedupe scenario from My Scenarios on your Dashboard. 

Your source, object, and record selection criteria will already be set for you. You can keep all settings or change them as you go.

Step 1: Select Records

Select your source file, database, or DupeBlocker under Select Source. CRM Object will use the organization your instance of DemandTools is linked to, while Import File will prompt you to upload a CSV file.

• You can use the Winner and Duplicate IDs export from a previous Dedupe scenario as your source file. (See Step 3: Merge Management below.)
• Selecting DupeBlocker allows you to select which object you want to see DupeBlocker scenarios for.
• For large jobs, check the box beside Bulk API. This option groups your records into larger batches, so it uses fewer API calls. Learn more about the Bulk API option .

Under the Object to Dedupe dropdown menu, select the object you want to find duplicates of. (You'll define what you consider to be a duplicate is in Step 2.)

• If you use DupeBlocker as your source, once you select an object, your DupeBlocker scenarios for that object will display.

In the Record Selection Criteria section, choose Add Selection Condition to narrow down the accounts included in your search. Learn more about Record Selection Criteria.

• In this example, set the Billing City to start with P.

• If you use DupeBlocker as your source, you will not be required to map fields or set criteria. Instead, a DupeBlocker Scenarios section will appear, showing Scenario Name, Duplicate Warnings, Potential Duplicates, Created By, Last Modified By, and Last Modified Date.

Fields to Show on Preview is an optional menu that lets you select the fields shown on your results page in step 3. If you don't use this menu, your results will include a subset of fields related to the dedupe object you selected, record criteria, and match conditions.

• If you use DupeBlocker as your source and don't use this menu, the preview grid will show the record ID as well as all fields that were used in the scenario's match conditions.

To move to step 2, choose Select Matching Fields.

• If you use DupeBlocker as your source, you will go directly to Preview. See Step 3: Merge Management.

Step 2: Match Records

In this step, you define what you consider to be duplicate records. Under the Matching Account Field dropdown, select the field you want to match on to find duplicates.

Select Comparison Type next, whether exact or more relaxed. A few examples include:

• Original Value: Use the original string value to compare.
• Cleaned Account Name: Use built-in cleanse rules to match similar names based on common punctuation, abbreviations, and common business prefixes. For example, The ABC Company and ABC Co. would be seen as matching account names.   These rules are customizable to meet your organization's specific needs.  To customize go to Settings - Manage Cleanse Rules. 
• Acronym: Use the first letter from each word in the field value.
• First X Letters: Use only part of the field value by specifying the number of characters to use.
• First X Words:
• Domain: Strip out all components of the field value except the domain portion.
• Address Cleanse: Normalize the address to ensure better matching between similar address field values.

A comprehensive list of comparison types including use cases and examples can be found here.

Four checkboxes let you adjust the logic for your search.

• Blank Values: Look for positive matches even when field values are blank for a given field. Please note that at least one field must have a non-blank match for a positive match to occur.
• Fuzzy Matching: Analyze words based on their phonetic value, removing vowels and analyzing the remaining consonants.
• Transposed Values: Analyze field values by reordering multiple words within each field value.
• Alpha Clean: Analyze field values by using only characters that are one of the 26 characters of the English alphabet, numbers 0-9, space, and ampersand.

• When you select this option and add one or more selection conditions, the results grid will only return duplicate groups where at least one of the records matches the criteria.
  • For example, you can filter to return all duplicates of an account where at least one record in the duplicate set has an external account number.
• You will still need to apply a Winner Rule to pick the desired record based on the criteria, but there will be fewer matches to evaluate, resulting in a quicker application of the Winner Rule.

• Matching on multiple fields acts as an AND operation. For example, if you define three field names, a record has to match on all three fields to be considered a duplicate.

To move to step 3, select Manage Merge.

• As Dedupe prepares your results, you can see how many records are being processed. If the results set is too large, you can cancel, then go back and narrow your search further.
• Moving to step 3 does not commit changes.

Step 3: Merge Management

In this step, you can review your duplicates, designated by brackets grouping records together, and select a winner for each bracket.

You can add additional fields to the preview grid without returning to step 1 by clicking the 3 lines in any column header, followed by clicking the 9 dots as pictured below:

Choosing a Winner

For small sets of duplicates, you can choose a winner manually. Select the trophy icon under the Winner column for the record you want to keep.

• Contacts who are Partner/Customer Portal or Community users are indicated with a purple highlight on the Full Name field. These users should always be merged with caution, and typically they should be chosen as the winning record.

For larger sets, automate the process with Winner Rules. Set your first rule, such as Most Recently Modified Record, and select Apply Winner Rules. Dedupe will use that rule to pick the winner record from each deduplicate group.

• Selecting the Add another Rule option lets you apply additional rules sequentially to catch more records.
• You can create new or edit existing rules. Learn how to create and apply winner rules.
• You can choose to apply winner rules to all records, selected records, or unselected records.

Setting your Merge Options tells Dedupe how to merge your winning records into your CRM.

• Prefix Records: For non-winning records, add a prefix instead of deleting.
• Task Creation: Track merge activity by creating a summary task or email, or track merges individually by creating a completed task per merge.
• Ownership Changes: Enable assignment rules to change the ownership of accounts, leads, or cases, based on any active assignment or territory during update or insert operations.
• Child Object Selection: Select which child objects will be included in your merge. Learn more about Child Object Selection.

Select the Use field rules checkbox to override specific fields from your Winner record with field values from the other records inside each duplicate group that you want to merge into the winning record. Learn how to create and apply field rules.

• For example, you can choose to update a specific field such as Name with the newest record, even if all other fields are updated with Most Recently Modified according to your Winner Rule.
• You can apply a field rule override to multiple fields of the same data type at once.

Selecting Fields to Retain

Dedupe uses different highlights to help you understand the fields being retained before the merge.

• Once a winner is selected, either manually or by winner rules, the winning record is highlighted in yellow.
• If field rules retain any value from a duplicate record, that field value is highlighted in green. This value will automatically override the winning record for that field.
• If a portal or community user is present on a record, the Full Name field is highlighted in purple. Typically, the record with a purple highlight should be chosen as the winner, but this is not automatic.

You can manually choose individual field values to keep in the winning record from across the records in that duplicate group.

1. Find the duplicate record that contains the value you want to keep. The field must be editable.
2. Right-click on the cell for that field.
3. Choose Select as winning field.
4. The field will turn green to show it is overriding the value from the winning record.

Deselecting Options

Different deselection workflows have different results. If you:

• Deselect a record in a duplicate group: Field rules are reapplied to the remaining records in the duplicate group.
• Change the field rule applied: Field rules are reapplied to duplicate groups with winning records selected.
• Deselect Use Field Rules with field rule applied: Update if Empty logic reapplied to duplicate groups with winning records selected.
• Make a change to a condition in an already applied field rule: Field rules are reapplied to the remaining records in the duplicate group.

Export Options

Before (or instead of) merging records, you can select Export Options.

• Export Grid Data: Download your results as displayed, with winner rows selected and duplicate rows intact.
• Winner and Duplicate IDs: Download just the Winner and Duplicate columns.
  • Winner: contains record IDs for the selected winning record from the applicable duplicate group.
  • Duplicate: contains record IDs for the records that will be merged into the winning record.

If you are not ready to merge records, you can export Winner and Duplicate IDs. Then later, when you are ready to merge, simply upload the export as your source file in a new Dedupe scenario instead of going through all the steps to recreate this one.

To finish, select Merge selected records for [x] duplicate groups. This action will commit changes to your CRM.

• In select circumstances, you can unmerge recent Dedupe scenarios from the Run History dashboard.
• Merges performed within a DupeBlocker workflow will show up as normal Dedupe scenarios in Run History and support Undo actions.
• Learn more about Run History.

If you want to re-run or schedule this scenario again in the future, open the Scenarios bar slider and select Save As, then name your scenario and select Save. 

• If you use DupeBlocker as your source, you can save the scenario configuration and automate it through the scheduler.