feat: support mongo connector #471

undertaker86001 · 2025-08-12T05:01:43Z

What does this PR do

Rationale for this change

Standards checklist

The PR title is descriptive
The commit messages are semantic
Necessary tests are added
Updated the release notes
Necessary documents have been added if this is a new feature
Performance tests checked, no obvious performance degradation

medcl · 2025-08-14T10:15:51Z

@sowen1023 please help to review this PR

- Add missing 'if' keyword in conditional statement at line 257 - Fixes parsing error that prevented tests from running Co-Authored-By: windWheel <[email protected]>

SteveLauC

Testing...

plugins/connectors/mongodb/transformer.go

sowen1023 · 2025-08-14T13:50:07Z

plugins/connectors/mongodb/transformer.go

+	count := 0
+	maxBatchSize := 1000 // Prevent memory overflow
+
+	for cursor.Next(context.Background()) && count < maxBatchSize {


Maybe we can use len(documents) instead of count to eliminate the separate variable?

plugins/connectors/mongodb/mongodb.md

docker/init-mongo.js

plugins/connectors/mongodb/utils.go

plugins/connectors/mongodb/config.go

sowen1023 · 2025-08-14T15:01:11Z

plugins/connectors/mongodb/utils.go

+	return nil
+}
+
+func (p *Plugin) shouldStop() bool {


shouldStop is never used and should not be used like this.

plugins/connectors/mongodb/connection.go

config/setup/zh-CN/mongodb.tpl

plugins/connectors/mongodb/plugin.go

plugins/connectors/mongodb/transformer.go

…erver into issue-456

plugins/connectors/mongodb/plugin.go

config/setup/zh-CN/connector.tpl

medcl · 2025-08-25T01:39:30Z

@undertaker86001 any updates of this PR?

undertaker86001 · 2025-08-25T07:25:55Z

@undertaker86001 any updates of this PR?

I'm resolving a conflict.

plugins/connectors/mongodb/config.go

web/src/pages/data-source/new/FieldMapping.tsx

medcl · 2025-09-01T07:08:48Z

web/src/components/datasource/type/index.jsx

@@ -17,6 +17,10 @@ export const Types = {
  RSS: 'rss',
  S3: 's3',
  Yuque: 'yuque',
+  S3: 's3',


duplicated key S3 here

also Confluence and NetworkDrive

medcl · 2025-09-01T07:14:35Z

plugins/connectors/mongodb/scanner.go:69:44: collConfig.TitleField undefined (type CollectionConfig has no field or method TitleField)
plugins/connectors/mongodb/scanner.go:69:75: collConfig.ContentField undefined (type CollectionConfig has no field or method ContentField)
plugins/connectors/mongodb/scanner.go:70:14: collConfig.CategoryField undefined (type CollectionConfig has no field or method CategoryField)
plugins/connectors/mongodb/scanner.go:70:48: collConfig.TagsField undefined (type CollectionConfig has no field or method TagsField)
plugins/connectors/mongodb/scanner.go:71:14: collConfig.URLField undefined (type CollectionConfig has no field or method URLField)
plugins/connectors/mongodb/scanner.go:71:43: collConfig.TimestampField undefined (type CollectionConfig has no field or method TimestampField)
plugins/connectors/mongodb/scanner.go:78:17: collConfig.TitleField undefined (type CollectionConfig has no field or method TitleField)
plugins/connectors/mongodb/scanner.go:79:59: collConfig.TitleField undefined (type CollectionConfig has no field or method TitleField)
plugins/connectors/mongodb/scanner.go:81:17: collConfig.ContentField undefined (type CollectionConfig has no field or method ContentField)
plugins/connectors/mongodb/scanner.go:82:59: collConfig.ContentField undefined (type CollectionConfig has no field or method ContentField)
plugins/connectors/mongodb/scanner.go:82:59: too many errors
make: *** [build] Error 1

medcl · 2025-09-01T07:19:47Z

I don’t think we need a dedicated sync manager or strategy specifically for MongoDB only. If such functionality is required, it should be built on top of the existing connector task infrastructure, and it is target for long term roadmap. For consistency and maintainability, why not keep it simple and align with how the other connectors work?

undertaker86001 · 2025-09-07T07:55:22Z

I don’t think we need a dedicated sync manager or strategy specifically for MongoDB only. If such functionality is required, it should be built on top of the existing connector task infrastructure, and it is target for long term roadmap. For consistency and maintainability, why not keep it simple and align with how the other connectors work?

OK, I'll make the changes

undertaker86001 added 2 commits August 12, 2025 12:59

mongo connector

d87d5d2

add docs

9aca2cb

undertaker86001 added 6 commits August 15, 2025 14:13

add mongodb frontend && update background

2f345d5

add test && doc

645931a

add en doc

d121461

remove useless doc

7d6dc99

add mutli mode

4ce84f4

update doc

fb634d8

undertaker86001 marked this pull request as ready for review August 16, 2025 00:22

fix: correct syntax error in MongoDB plugin tests

f5e66b4

- Add missing 'if' keyword in conditional statement at line 257 - Fixes parsing error that prevented tests from running Co-Authored-By: windWheel <[email protected]>

SteveLauC reviewed Aug 19, 2025

View reviewed changes

sowen1023 reviewed Aug 19, 2025

View reviewed changes

config/setup/zh-CN/mongodb.tpl Outdated Show resolved Hide resolved

undertaker86001 added 5 commits August 19, 2025 20:24

fix connection leak

cc6547f

remove time.sleep

f407c4c

refactor: support dynamic lastsynctime

f793bf9

refactor:adapter sync_strategy

45716f8

refactor: pre-allocate slice

f28b36f

medcl reviewed Aug 20, 2025

View reviewed changes

config/setup/zh-CN/mongodb.tpl Outdated Show resolved Hide resolved

medcl reviewed Aug 20, 2025

View reviewed changes

plugins/connectors/mongodb/plugin.go Outdated Show resolved Hide resolved

medcl reviewed Aug 20, 2025

View reviewed changes

plugins/connectors/mongodb/transformer.go Outdated Show resolved Hide resolved

undertaker86001 added 6 commits August 21, 2025 13:35

add system field

dec2283

refactor: use task framework

c0ac8aa

Merge branch 'issue-456' of https://github.com/undertaker86001/coco-s…

5e75ebe

…erver into issue-456

remove useless files

4d817e8

remove useless files

fe69e19

update doc

bb3351a

medcl reviewed Aug 22, 2025

View reviewed changes

plugins/connectors/mongodb/plugin.go Outdated Show resolved Hide resolved

medcl reviewed Aug 22, 2025

View reviewed changes

config/setup/zh-CN/connector.tpl Outdated Show resolved Hide resolved

undertaker86001 added 2 commits August 22, 2025 11:18

remove useless doc

df53181

refactor: simpfy monitor

23f0111

undertaker86001 and others added 5 commits August 25, 2025 15:31

extract common config

7114911

Merge remote-tracking branch 'upstream/main' into issue-456

7f1e583

merge conf

dc5ae54

fix test && remove useless code

9aa28fb

fix imports

ae52455

medcl reviewed Aug 26, 2025

View reviewed changes

plugins/connectors/mongodb/config.go Outdated Show resolved Hide resolved

medcl reviewed Aug 26, 2025

View reviewed changes

web/src/pages/data-source/new/FieldMapping.tsx Outdated Show resolved Hide resolved

undertaker86001 added 3 commits August 27, 2025 07:51

add field mapping

eeeb02e

update imports

ebb21ef

Merge remote-tracking branch 'upstream/main' into issue-456

8b7ff74

medcl reviewed Sep 1, 2025

View reviewed changes

feat: support mongo connector #471

Are you sure you want to change the base?

feat: support mongo connector #471

Conversation

undertaker86001 commented Aug 12, 2025

What does this PR do

Rationale for this change

Standards checklist

Uh oh!

medcl commented Aug 14, 2025

Uh oh!

SteveLauC left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sowen1023 Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sowen1023 Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

medcl commented Aug 25, 2025

Uh oh!

undertaker86001 commented Aug 25, 2025

Uh oh!

Uh oh!

Uh oh!

medcl Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

medcl Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

medcl commented Sep 1, 2025

Uh oh!

medcl commented Sep 1, 2025

Uh oh!

undertaker86001 commented Sep 7, 2025

Uh oh!

Uh oh!