Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Export accounts from workspace to csv and add items from drag-n-drop csv #184

Merged
merged 7 commits into from
Dec 9, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion components/repositories/RepositoriesTable.tsx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import Link from 'next/link'
import { UserGroupIcon } from '@heroicons/react/20/solid'
import { formatDistanceToNow } from 'date-fns'
import { AppBskyActorProfile, ComAtprotoAdminDefs } from '@atproto/api'
import { AppBskyActorProfile } from '@atproto/api'
import { Repo } from '@/lib/types'
import { LoadMoreButton } from '../common/LoadMoreButton'
import { ReviewStateIcon } from '@/subject/ReviewStateMarker'
Expand Down
31 changes: 27 additions & 4 deletions components/workspace/List.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,18 @@ import { StatusBySubject } from '@/subject/useSubjectStatus'
import { SubjectOverview } from '@/reports/SubjectOverview'
import { ReviewStateIcon } from '@/subject/ReviewStateMarker'
import { PreviewCard } from '@/common/PreviewCard'
import { useWorkspaceExportMutation } from './hooks'

interface WorkspaceListProps {
list: string[]
subjectStatuses: StatusBySubject
onRemoveItem: (item: string) => void
}

const GroupTitles = {
dids: 'Accounts',
}

const WorkspaceList: React.FC<WorkspaceListProps> = ({
list,
subjectStatuses,
Expand All @@ -28,15 +33,19 @@ const WorkspaceList: React.FC<WorkspaceListProps> = ({
return (
<div>
<div className="space-y-2">
{Object.entries(groupedItems).map(([key, items], parentIndex) => {
{Object.entries(groupedItems).map(([key, items]) => {
if (!items.length) return null
return (
<ListGroup
key={key}
items={items}
subjectStatuses={subjectStatuses}
onRemoveItem={onRemoveItem}
title={`${key.charAt(0).toUpperCase()}${key.slice(1)}`}
canExport={key === 'dids'}
title={
GroupTitles[key] ||
`${key.charAt(0).toUpperCase()}${key.slice(1)}`
}
/>
)
})}
Expand All @@ -50,14 +59,17 @@ const ListGroup = ({
title,
subjectStatuses,
onRemoveItem,
canExport,
}: {
items: string[]
title: string
canExport?: boolean
} & Omit<WorkspaceListProps, 'list'>) => {
const [lastCheckedIndex, setLastCheckedIndex] = useState<number | null>(null)
const checkboxesRef = useRef<(HTMLInputElement | null)[]>([])
const [detailShown, setDetailShown] = useState<string[]>([])
const areAllDetailShown = items.every((item) => detailShown.includes(item))
const exportMutation = useWorkspaceExportMutation()

// This ensures that when shift+clicking checkboxes, all checkboxes between the last interacted item are toggled
const handleChange = (
Expand All @@ -79,12 +91,23 @@ const ListGroup = ({
}

return (
<div className="pb-2">
<div className="py-2">
<div className="flex justify-between mb-1 mr-2">
<h5 className="text-base font-semibold">
{title}({items.length})
</h5>
<div>
<div className="flex gap-1">
{canExport && (
<ActionButton
size="sm"
appearance="outlined"
onClick={() => exportMutation.mutateAsync(items)}
>
<span className="text-xs">
{exportMutation.isLoading ? 'Exporting...' : 'Export'}
</span>
</ActionButton>
)}
<ActionButton
size="sm"
appearance="outlined"
Expand Down
76 changes: 75 additions & 1 deletion components/workspace/hooks.tsx
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
import { createCSV, downloadCSV } from '@/lib/csv'
import { getLocalStorageData, setLocalStorageData } from '@/lib/local-storage'
import { pluralize } from '@/lib/util'
import { buildBlueSkyAppUrl, chunkArray, pluralize } from '@/lib/util'
import { useLabelerAgent } from '@/shell/ConfigurationContext'
import { AppBskyActorProfile, ToolsOzoneModerationDefs } from '@atproto/api'
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
import { useRef } from 'react'
import { toast } from 'react-toastify'
Expand Down Expand Up @@ -95,6 +98,77 @@ export const useWorkspaceEmptyMutation = () => {
return mutation
}

export const useWorkspaceExportMutation = () => {
const labelerAgent = useLabelerAgent()
return useMutation({
mutationFn: async (items: string[]) => {
// Items are exported in groups so we can expect all items in the group to be of same type
// For now, only support exporting accounts
if (!items[0].startsWith('did:')) {
toast.error(`Exporting is only enabled for accounts.`)
return []
}

const data: Record<
string,
ToolsOzoneModerationDefs.RepoViewDetail | null
> = {}

for (const itemChunk of chunkArray(items, 50)) {
await Promise.all(
itemChunk.map(async (did) => {
try {
const { data: repo } =
await labelerAgent.tools.ozone.moderation.getRepo({
did,
})

data[did] = repo
} catch (error) {
// For now we're just swallowing errors and exporting what I can
console.error(error)
data[did] = null
}
}),
)
}

downloadCSV(
createCSV({
headers: [
'did',
'handle',
'email',
'ip',
'name',
'labels',
'profile',
],
lines: Object.values(data).map((repo) => {
if (!repo) return ''
const profile = AppBskyActorProfile.isRecord(repo.relatedRecords[0])
? (repo.relatedRecords[0] as AppBskyActorProfile.Record)
: null

const line: string[] = [
repo.did,
repo.handle,
repo.email || 'Unknown',
`${repo.ip || 'Unknown'}`,
`${profile?.displayName || 'Unknown'}`,
repo.labels?.map(({ val }) => val).join(', ') || 'None',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I almost wonder if an empty value could be more useful than a default value here. I suppose it depends how the CSV will be used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I think our current use case is for mod folks to get these exports and review manually in which case, I feel like the explicit default values are more helpful.

buildBlueSkyAppUrl({ did: repo.did }),
]
return line.join(',')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Values containing commas or newlines will end up breaking the CSV format. If we add some escaping we should be able to address it, though:

const escapeCSVValue = (value: string) => {
  if (value.contains(',') || value.contains('"') || value.contains('\r') || value.contains('\n')) {
    return `"${value.replaceAll('"', '""')}"`
  }
  return value
}

// ...

  return line.map(escapeCSVValue).join(',')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yh except for labels/tags list, we don't have anything that contain those characters though but will add to make it more future proof.

}),
}),
)

return data
},
})
}

const getList = (): string[] => {
const list = getLocalStorageData<string>(WORKSPACE_LIST_KEY)
if (!list) return []
Expand Down
39 changes: 39 additions & 0 deletions lib/csv.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
export type CsvContent = {
filename: string
headerRow: string
body: string
}

export function createCSV({
headers,
lines,
filename,
lineDelimiter = '\n',
}: {
lines: string[]
headers: string[]
filename?: string
lineDelimiter?: string
}) {
return {
filename: (filename || Date.now().toString()) + '.csv',
headerRow: headers.join(',') + lineDelimiter, // make your own csv head
body: lines.join(lineDelimiter),
}
}
Copy link
Contributor

@matthieusieben matthieusieben Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return value does not match the returned type. Is there a particular reason for that ?

The data structure to represent the CSV here is a mix between encoded data (a multi line string of comma separated values, with the firt line being a header) and "parsed" data (internal data structure), Looks like we could benefit from defining and using a type interface (or even a class ?) for internal representation of CSVs:

type CsvContent = {
  headers: string[]
  rows: [][]
  lineDelimiter: "," | ":" | "\t" 
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The return value does not match the returned type

not sure if I follow? the return type is expected to be CsvContent which i left implicit I guess. agreed we can tidy this up a bit more to make a more strict definition but for now the use-case is super low-stake so wanted to keep this flexible/open.


export function downloadCSV(csv: CsvContent) {
var csvContent = csv.headerRow + csv.body
if (!csvContent.match(/^data:text\/csv/i)) {
csvContent = 'data:text/csv;charset=utf-8,' + csvContent // use 'data:text/csv;charset=utf-8,\ufeff', if you consider using the excel
}
var data = encodeURI(csvContent)

var link = document.createElement('a')
link.href = data
link.download = csv.filename

document.body.appendChild(link)
link.click()
document.body.removeChild(link)
}
Loading