Skip to content

Templates and examples 模板和示例#

以下是 Remove Duplicates 去重节点 的一些模板和示例。

Continuous examples 连续示例

本节中包含的示例是一个序列。请按顺序从一个示例到另一个示例，以避免意外结果。

Templates 模板#

Browse 模板和示例 integration templates, or search all templates

Set up sample data using the Code node 使用Code节点设置示例数据#

创建一个包含一些示例输入数据的工作流来尝试 Remove Duplicates 去重节点。

  1. 将 Code 代码节点添加到画布并将其连接到 Manual Trigger 手动触发器节点。
  2. 在 Code 代码节点中，将 Mode 模式 设置为 Run Once for Each Item 为每个项目运行一次，将 Language 语言 设置为 JavaScript。
  3. 将以下 JavaScript 代码片段粘贴到 JavaScript 字段中： 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
    let data =[];

return {
  data: [
    { id: 1, name: 'Taylor Swift', job: 'Pop star', last_updated: '2024-09-20T10:12:43.493Z' },
    { id: 2, name: 'Ed Sheeran', job: 'Singer-songwriter', last_updated: '2024-10-05T08:30:59.493Z' },
    { id: 3, name: 'Adele', job: 'Singer-songwriter', last_updated: '2024-10-07T14:15:59.493Z' },
    { id: 4, name: 'Bruno Mars', job: 'Singer-songwriter', last_updated: '2024-08-25T17:45:12.493Z' },
    { id: 1, name: 'Taylor Swift', job: 'Pop star', last_updated: '2024-09-20T10:12:43.493Z' },  // duplicate
    { id: 5, name: 'Billie Eilish', job: 'Singer-songwriter', last_updated: '2024-09-10T09:30:12.493Z' },
    { id: 6, name: 'Katy Perry', job: 'Pop star', last_updated: '2024-10-08T12:30:45.493Z' },
    { id: 2, name: 'Ed Sheeran', job: 'Singer-songwriter', last_updated: '2024-10-05T08:30:59.493Z' },  // duplicate
    { id: 7, name: 'Lady Gaga', job: 'Pop star', last_updated: '2024-09-15T14:45:30.493Z' },
    { id: 8, name: 'Rihanna', job: 'Pop star', last_updated: '2024-10-01T11:50:22.493Z' },
    { id: 3, name: 'Adele', job: 'Singer-songwriter', last_updated: '2024-10-07T14:15:59.493Z' },  // duplicate
    //{ id: 9, name: 'Tom Hanks', job: 'Actor', last_updated: '2024-10-17T13:58:31.493Z' },
    //{ id: 0, name: 'Madonna', job: 'Pop star', last_updated: '2024-10-17T17:11:38.493Z' },
    //{ id: 15, name: 'Bob Dylan', job: 'Folk singer', last_updated: '2024-09-24T08:03:16.493Z'},
    //{ id: 10, name: 'Harry Nilsson', job: 'Singer-songwriter', last_updated: '2020-10-17T17:11:38.493Z' },
    //{ id: 11, name: 'Kylie Minogue', job: 'Pop star', last_updated: '2024-10-24T08:03:16.493Z'},
  ]
}
  4. 将 Split Out 拆分节点添加到画布并将其连接到 Code 代码节点。
  5. 在 Split Out 拆分节点中，在 Fields To Split Out 要拆分的字段 字段中输入 data。

Removing duplicates from the current input 删除当前输入中的重复项#

  1. 将 Remove Duplicates 去重节点添加到画布并将其连接到 Split Out 拆分节点。选择 Remove items repeated within current input 删除当前输入中重复的项目 作为 Action 操作 开始。
  2. 打开 Remove Duplicates 去重节点并确保 Operation 操作 设置为 Remove Items Repeated Within Current Input 删除当前输入中重复的项目。
  3. 在 Compare 比较 字段中选择 All fields 所有字段。
  4. 选择 Execute step 执行步骤 运行 Remove Duplicates 去重节点，删除当前输入中的重复数据。

n8n removes the items that have the same data across all fields. Your output in table view should look like this:

id name job last_updated
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z
  1. Open the Remove Duplicates node again and change the Compare parameter to Selected Fields.
  2. In the Fields To Compare field, enter job.
  3. Select Execute step to run the Remove Duplicates node, removing duplicated data in the current input.

n8n removes the items in the current input that have the same job data. Your output in table view should look like this:

id name job last_updated
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z

Keep items where the value is new#

  1. Open the Remove Duplicates node and set the Operation to Remove Items Processed in Previous Executions.
  2. Set the Keep Items Where parameter to Value Is New.
  3. Set the Value to Dedupe On parameter to {{ $json.name }}.
  4. On the canvas, select Execute workflow to run the workflow. Open the Remove Duplicates node to examine the results.

n8n compares the current input data to the items stored from previous executions. Since this is the first time running the Remove Duplicates node with this operation, n8n processes all data items and places them into the Kept output tab. The order of the items may be different than the order in the input data:

id name job last_updated
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z

Items are only compared against previous executions

The current input items are only compared against the stored items from previous executions. This means that items repeated within the current input aren't removed in this mode of operation. If you need to remove duplicate items within the current input and across executions, connect two Remove Duplicate nodes together sequentially. Set the first to use the Remove Items Repated Within Current Input operation and the second to use the Remove Items Processed in Previous Executions operation.

  1. Open the Code node and uncomment (remove the // from) the line for "Tom Hanks."
  2. On the canvas, select Execute workflow again. Open the Remove Duplicates node again to examine the results.

n8n compares the current input data to the items stored from previous executions. This time, the Kept tab contains the one new record from the Code node:

id name job last_updated
9 Tom Hanks Actor 2024-10-17T13:58:31.493Z

The Discarded tab contains the items processed by the previous execution:

id name job last_updated
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z

Before continuing, clear the duplication history to get ready for the next example:

  1. Open the Remove Duplicates node and set the Operation to Clear Deduplication History.
  2. Select Execute step to clear the current duplication history.

Keep items where the value is higher than any previous value#

  1. Open the Remove Duplicates node and set the Operation to Remove Items Processed in Previous Executions.
  2. Set the Keep Items Where parameter to Value Is Higher than Any Previous Value.
  3. Set the Value to Dedupe On parameter to {{ $json.id }}.
  4. On the canvas, select Execute workflow to run the workflow. Open the Remove Duplicates node to examine the results.

n8n compares the current input data to the items stored from previous executions. Since this is the first time running the Remove Duplicates node after clearing the history, n8n processes all data items and places them into the Kept output tab. The order of the items may be different than the order in the input data:

id name job last_updated
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z
9 Tom Hanks Actor 2024-10-17T13:58:31.493Z
  1. Open the Code node and uncomment (remove the // from) the lines for "Madonna" and "Bob Dylan."
  2. On the canvas, select Execute workflow again. Open the Remove Duplicates node again to examine the results.

n8n compares the current input data to the items stored from previous executions. This time, the Kept tab contains a single entry for "Bob Dylan." n8n keeps this item because its id column value (15) is higher than any previous values (the previous maximum value was 9):

id name job last_updated
15 Bob Dylan Folk singer 2024-09-24T08:03:16.493Z

The Discarded tab contains the 13 items with an id column value equal to or less than the previous maximum value (9). Even though it's new, this table includes the entry for "Madonna" because its id value isn't larger than the previous maximum value:

id name job last_updated
0 Madonna Pop star 2024-10-17T17:11:38.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z
9 Tom Hanks Actor 2024-10-17T13:58:31.493Z

Before continuing, clear the duplication history to get ready for the next example:

  1. Open the Remove Duplicates node and set the Operation to Clear Deduplication History.
  2. Select Execute step to clear the current duplication history.

Keep items where the value is a date later than any previous date#

  1. Open the Remove Duplicates node and set the Operation to Remove Items Processed in Previous Executions.
  2. Set the Keep Items Where parameter to Value Is a Date Later than Any Previous Date.
  3. Set the Value to Dedupe On parameter to {{ $json.last_updated }}.
  4. On the canvas, select Execute workflow to run the workflow. Open the Remove Duplicates node to examine the results.

n8n compares the current input data to the items stored from previous executions. Since this is the first time running the Remove Duplicates node after clearing the history, n8n processes all data items and places them into the Kept output tab. The order of the items may be different than the order in the input data:

id name job last_updated
0 Madonna Pop star 2024-10-17T17:11:38.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z
9 Tom Hanks Actor 2024-10-17T13:58:31.493Z
15 Bob Dylan Folk singer 2024-09-24T08:03:16.493Z
  1. Open the Code node and uncomment (remove the // from) the lines for "Harry Nilsson" and "Kylie Minogue."
  1. On the canvas, select Execute workflow again. Open the Remove Duplicates node again to examine the results.

n8n compares the current input data to the items stored from previous executions. This time, the Kept tab contains a single entry for "Kylie Minogue." n8n keeps this item because its last_updated column value (2024-10-24T08:03:16.493Z) is later than any previous values (the previous latest date was 2024-10-17T17:11:38.493Z):

id name job last_updated
11 Kylie Minogue Pop star 2024-10-24T08:03:16.493Z

The Discarded tab contains the 15 items with a last_updated column value equal to or earlier than the previous latest date (2024-10-17T17:11:38.493Z). Even though it's new, this table includes the entry for "Harry Nilsson" because its last_updated value isn't later than the previous maximum value:

id name job last_updated
10 Harry Nilsson Singer-songwriter 2020-10-17T17:11:38.493Z
0 Madonna Pop star 2024-10-17T17:11:38.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
1 Taylor Swift Pop star 2024-09-20T10:12:43.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
2 Ed Sheeran Singer-songwriter 2024-10-05T08:30:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
3 Adele Singer-songwriter 2024-10-07T14:15:59.493Z
4 Bruno Mars Singer-songwriter 2024-08-25T17:45:12.493Z
5 Billie Eilish Singer-songwriter 2024-09-10T09:30:12.493Z
6 Katy Perry Pop star 2024-10-08T12:30:45.493Z
7 Lady Gaga Pop star 2024-09-15T14:45:30.493Z
8 Rihanna Pop star 2024-10-01T11:50:22.493Z
9 Tom Hanks Actor 2024-10-17T13:58:31.493Z
15 Bob Dylan Folk singer 2024-09-24T08:03:16.493Z