Updated script for better splitting of synonyms

This commit is contained in:
Jamie Hardt committed 2024-10-30 10:51:05 -07:00
1 parent 485c5175b8
commit b21e80f708
17 files changed
+11401 -27093

No files matched your search

+341 -733
View File
File diff suppressed because it is too large. Load diff
+270 -595
View File
File diff suppressed because it is too large. Load diff
+211 -423
View File
File diff suppressed because it is too large. Load diff
+942 -2489
View File
File diff suppressed because it is too large. Load diff
+499 -1058
View File
File diff suppressed because it is too large. Load diff
+1117 -2972
View File
File diff suppressed because it is too large. Load diff
+942 -2494
View File
File diff suppressed because it is too large. Load diff
+418 -882
View File
File diff suppressed because it is too large. Load diff
+280 -598
View File
File diff suppressed because it is too large. Load diff
+943 -2081
View File
File diff suppressed because it is too large. Load diff
+1018 -2732
View File
File diff suppressed because it is too large. Load diff
+1024 -2760
View File
File diff suppressed because it is too large. Load diff
+946 -2050
View File
File diff suppressed because it is too large. Load diff
+423 -926
View File
File diff suppressed because it is too large. Load diff
+1127 -2359
View File
File diff suppressed because it is too large. Load diff
+896 -1940
View File
File diff suppressed because it is too large. Load diff
+4 -1
View File
@@ -80,7 +80,10 @@ for lang in langs:
syns_raw = category['Synonyms']
assert type(syns_raw) == str, \
f"Synonym list (lang: {lang}, {category['CatID']}) was not readable"
syn_list = re.split(r'\W+', syns_raw)
split_pattern = r',\s*'
if lang in ['zh','ar','kr','ja','tw']:
split_pattern = r'\W+'
syn_list = re.split(split_pattern, syns_raw)
category['Synonyms'] = [s.lower() for s in syn_list]
schedule.append(category)