Skip to content

Instantly share code, notes, and snippets.

@DaveKin
Last active October 16, 2024 22:01
Show Gist options
  • Save DaveKin/0de68255a7d7f8fa85f6008aa3ce26b2 to your computer and use it in GitHub Desktop.
Save DaveKin/0de68255a7d7f8fa85f6008aa3ce26b2 to your computer and use it in GitHub Desktop.
SQL function to remove duplicate entries from a JSON array. Based on MYSQL 5.7+
CREATE FUNCTION JSON_UNIQ(arr JSON) RETURNS json
BEGIN
SET @arr = arr;
SET @a_length = JSON_LENGTH(@arr);
SET @loop_index = @a_length;
WHILE @loop_index >= 0 DO
SET @item = JSON_UNQUOTE(JSON_EXTRACT(@arr, concat('$[',@loop_index,']')));
SET @itemcount = coalesce(JSON_LENGTH(JSON_SEARCH(@arr, 'all', @item)), 0);
IF @itemcount > 1 THEN
SET @arr = JSON_REMOVE(@arr, CONCAT('$[',@loop_index,']'));
SET @loop_index = @loop_index - 1;
END IF;
SET @loop_index = @loop_index - 1;
End WHILE;
RETURN CAST(@arr AS JSON);
END
@AlexanderHentzsch
Copy link

AlexanderHentzsch commented Mar 31, 2023

Since I got the error

"ERROR 1418: This function has none of DETERMINISTIC, NO SQL, or READS SQL DATA in its declaration and binary logging is enabled (you might want to use the less safe log_bin_trust_function_creators variable)"

when creating the function, and also not all duplicates were removed but only one, I asked ChatGPT. Here is his answer:

CREATE FUNCTION JSON_UNIQ(arr JSON) RETURNS json READS SQL DATA
BEGIN
    DECLARE tempArr JSON DEFAULT JSON_ARRAY();
    DECLARE tempItem JSON DEFAULT NULL;
    DECLARE tempIndex INT DEFAULT NULL;
    DECLARE i INT DEFAULT 0;
    
    WHILE i < JSON_LENGTH(arr) DO
        SET tempItem = JSON_EXTRACT(arr, CONCAT('$[', i, ']'));

        IF NOT JSON_CONTAINS(tempArr, tempItem) THEN
            SET tempArr = JSON_INSERT(tempArr, CONCAT('$[', JSON_LENGTH(tempArr), ']'), tempItem);
        END IF;

        SET i = i + 1;
    END WHILE;

    RETURN tempArr;
END;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment